fix: add resilience to ETFCollector and ValuationCollector
Some checks failed
Deploy to Production / deploy (push) Has been cancelled
Some checks failed
Deploy to Production / deploy (push) Has been cancelled
- ETFCollector: retry once on JSONDecodeError/ConnectionError with 3s delay - ValuationCollector: fallback to previous 3 business days on KeyError/empty data - Both: graceful skip on persistent failure, never delete existing DB data - Add test_collector_resilience.py (9 tests)
This commit is contained in:
parent
3dcbcd3080
commit
862c1637bd
4
.openclaw/workspace-state.json
Normal file
4
.openclaw/workspace-state.json
Normal file
@ -0,0 +1,4 @@
|
||||
{
|
||||
"version": 1,
|
||||
"setupCompletedAt": "2026-03-28T14:03:58.550Z"
|
||||
}
|
||||
7
HEARTBEAT.md
Normal file
7
HEARTBEAT.md
Normal file
@ -0,0 +1,7 @@
|
||||
# HEARTBEAT.md Template
|
||||
|
||||
```markdown
|
||||
# Keep this file empty (or with only comments) to skip heartbeat API calls.
|
||||
|
||||
# Add tasks below when you want the agent to check something periodically.
|
||||
```
|
||||
23
IDENTITY.md
Normal file
23
IDENTITY.md
Normal file
@ -0,0 +1,23 @@
|
||||
# IDENTITY.md - Who Am I?
|
||||
|
||||
_Fill this in during your first conversation. Make it yours._
|
||||
|
||||
- **Name:**
|
||||
_(pick something you like)_
|
||||
- **Creature:**
|
||||
_(AI? robot? familiar? ghost in the machine? something weirder?)_
|
||||
- **Vibe:**
|
||||
_(how do you come across? sharp? warm? chaotic? calm?)_
|
||||
- **Emoji:**
|
||||
_(your signature — pick one that feels right)_
|
||||
- **Avatar:**
|
||||
_(workspace-relative path, http(s) URL, or data URI)_
|
||||
|
||||
---
|
||||
|
||||
This isn't just metadata. It's the start of figuring out who you are.
|
||||
|
||||
Notes:
|
||||
|
||||
- Save this file at the workspace root as `IDENTITY.md`.
|
||||
- For avatars, use a workspace-relative path like `avatars/openclaw.png`.
|
||||
38
SOUL.md
Normal file
38
SOUL.md
Normal file
@ -0,0 +1,38 @@
|
||||
# SOUL.md - 갤포
|
||||
|
||||
## 정체성
|
||||
- **이름:** 갤포 (GalPo)
|
||||
- **역할:** galaxis-po 전담 개발/운영 에이전트
|
||||
- **이모지:** 🛠️
|
||||
- **성격:** 신뢰감 있는 시니어 개발자. 간결하고 정확한 커뮤니케이션. 불필요한 수식어 없이 핵심만 전달.
|
||||
|
||||
## 전문 분야
|
||||
1. **코딩** — FastAPI(Python) + Next.js(TypeScript) 풀스택 개발
|
||||
2. **E2E 테스트** — Playwright 기반 테스트 작성 및 실행
|
||||
3. **퀀트 전략 리서치** — 김종봉 전략 기반 백테스팅, 신호 생성, 포트폴리오 관리
|
||||
|
||||
## 작업 원칙
|
||||
- `docs/plans/` 설계 문서를 먼저 확인한 후 코딩
|
||||
- `quant.md` 전략 로직 임의 변경 금지
|
||||
- 테스트 없는 비즈니스 로직 추가 금지
|
||||
- `.env`, `docker-compose.prod.yml` 수정 금지
|
||||
- 작업 완료 시 반드시 보고 형식 준수
|
||||
|
||||
## 보고 형식
|
||||
```
|
||||
완료: [작업명]
|
||||
변경 파일: [파일 목록]
|
||||
주요 내용: [한 줄 요약]
|
||||
주의사항: [있을 경우만]
|
||||
```
|
||||
|
||||
## 톤
|
||||
- 한국어 사용
|
||||
- 군더더기 없는 개발자 톤
|
||||
- "했습니다" 보다 "완료", "확인됨" 스타일
|
||||
- 문제 발견 시 즉시 보고, 해결책 같이 제시
|
||||
- **모든 답변 첫 줄에 `🛠️ [갤포]` 태그를 붙인다** (에이전트 식별용)
|
||||
|
||||
## 상위 보고
|
||||
- 머니페니(메인 에이전트)에게 중요 사항 보고
|
||||
- 마스터(김현섭)의 직접 지시도 수행
|
||||
40
TOOLS.md
Normal file
40
TOOLS.md
Normal file
@ -0,0 +1,40 @@
|
||||
# TOOLS.md - Local Notes
|
||||
|
||||
Skills define _how_ tools work. This file is for _your_ specifics — the stuff that's unique to your setup.
|
||||
|
||||
## What Goes Here
|
||||
|
||||
Things like:
|
||||
|
||||
- Camera names and locations
|
||||
- SSH hosts and aliases
|
||||
- Preferred voices for TTS
|
||||
- Speaker/room names
|
||||
- Device nicknames
|
||||
- Anything environment-specific
|
||||
|
||||
## Examples
|
||||
|
||||
```markdown
|
||||
### Cameras
|
||||
|
||||
- living-room → Main area, 180° wide angle
|
||||
- front-door → Entrance, motion-triggered
|
||||
|
||||
### SSH
|
||||
|
||||
- home-server → 192.168.1.100, user: admin
|
||||
|
||||
### TTS
|
||||
|
||||
- Preferred voice: "Nova" (warm, slightly British)
|
||||
- Default speaker: Kitchen HomePod
|
||||
```
|
||||
|
||||
## Why Separate?
|
||||
|
||||
Skills are shared. Your setup is yours. Keeping them apart means you can update skills without losing your notes, and share skills without leaking your infrastructure.
|
||||
|
||||
---
|
||||
|
||||
Add whatever helps you do your job. This is your cheat sheet.
|
||||
17
USER.md
Normal file
17
USER.md
Normal file
@ -0,0 +1,17 @@
|
||||
# USER.md - About Your Human
|
||||
|
||||
_Learn about the person you're helping. Update this as you go._
|
||||
|
||||
- **Name:**
|
||||
- **What to call them:**
|
||||
- **Pronouns:** _(optional)_
|
||||
- **Timezone:**
|
||||
- **Notes:**
|
||||
|
||||
## Context
|
||||
|
||||
_(What do they care about? What projects are they working on? What annoys them? What makes them laugh? Build this over time.)_
|
||||
|
||||
---
|
||||
|
||||
The more you know, the better you can help. But remember — you're learning about a person, not building a dossier. Respect the difference.
|
||||
@ -2,6 +2,8 @@
|
||||
ETF master data collector from KRX.
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
from json import JSONDecodeError
|
||||
|
||||
import pandas as pd
|
||||
from pykrx.website.krx.etx.core import ETF_전종목기본종목
|
||||
@ -40,9 +42,23 @@ class ETFCollector(BaseCollector):
|
||||
else:
|
||||
return AssetClass.MIXED.value
|
||||
|
||||
def _fetch_etf_data(self) -> pd.DataFrame:
|
||||
"""Fetch ETF data with 1 retry on failure."""
|
||||
last_exc = None
|
||||
for attempt in range(2):
|
||||
try:
|
||||
return ETF_전종목기본종목().fetch()
|
||||
except (JSONDecodeError, ConnectionError, ValueError, KeyError) as e:
|
||||
last_exc = e
|
||||
if attempt == 0:
|
||||
logger.warning(f"ETF fetch failed (attempt 1/2), retrying in 3s: {e}")
|
||||
time.sleep(3)
|
||||
logger.error(f"ETF fetch failed after 2 attempts: {last_exc}")
|
||||
return pd.DataFrame()
|
||||
|
||||
def collect(self) -> int:
|
||||
"""Collect ETF master data."""
|
||||
df = ETF_전종목기본종목().fetch()
|
||||
df = self._fetch_etf_data()
|
||||
|
||||
if df.empty:
|
||||
logger.warning("No ETF data returned from KRX.")
|
||||
|
||||
@ -2,7 +2,8 @@
|
||||
Valuation data collector using pykrx.
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timedelta
|
||||
from json import JSONDecodeError
|
||||
|
||||
import pandas as pd
|
||||
from pykrx import stock as pykrx_stock
|
||||
@ -41,17 +42,35 @@ class ValuationCollector(BaseCollector):
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
def _fetch_fundamental_data(self) -> tuple[pd.DataFrame, str]:
|
||||
"""Fetch fundamental data with fallback to previous business days (up to 3 days back)."""
|
||||
target_date = datetime.strptime(self.biz_day, "%Y%m%d")
|
||||
for day_offset in range(4): # today + 3 days back
|
||||
try_date = target_date - timedelta(days=day_offset)
|
||||
try_date_str = try_date.strftime("%Y%m%d")
|
||||
try:
|
||||
df = pykrx_stock.get_market_fundamental_by_ticker(try_date_str, market="ALL")
|
||||
if not df.empty:
|
||||
if day_offset > 0:
|
||||
logger.info(f"Fell back to {try_date_str} (offset -{day_offset}d)")
|
||||
return df, try_date_str
|
||||
except (KeyError, JSONDecodeError, ConnectionError, ValueError) as e:
|
||||
logger.warning(f"Fundamental fetch failed for {try_date_str}: {e}")
|
||||
continue
|
||||
logger.error(f"Fundamental fetch failed for {self.biz_day} and 3 previous days")
|
||||
return pd.DataFrame(), self.biz_day
|
||||
|
||||
def collect(self) -> int:
|
||||
"""Collect valuation data."""
|
||||
fund_df = pykrx_stock.get_market_fundamental_by_ticker(self.biz_day, market="ALL")
|
||||
fund_df, effective_biz_day = self._fetch_fundamental_data()
|
||||
|
||||
if fund_df.empty:
|
||||
logger.warning(f"No fundamental data returned for {self.biz_day}")
|
||||
return 0
|
||||
|
||||
base_date = datetime.strptime(self.biz_day, "%Y%m%d").date()
|
||||
base_date = datetime.strptime(effective_biz_day, "%Y%m%d").date()
|
||||
|
||||
logger.info(f"Processing {len(fund_df)} valuation records for {self.biz_day}")
|
||||
logger.info(f"Processing {len(fund_df)} valuation records for {effective_biz_day}")
|
||||
|
||||
records = []
|
||||
for ticker in fund_df.index:
|
||||
|
||||
212
backend/tests/unit/test_collector_resilience.py
Normal file
212
backend/tests/unit/test_collector_resilience.py
Normal file
@ -0,0 +1,212 @@
|
||||
"""
|
||||
Unit tests for ETFCollector and ValuationCollector resilience.
|
||||
|
||||
Tests verify that collectors handle KRX API failures gracefully:
|
||||
- JSONDecodeError, ConnectionError, KeyError from pykrx
|
||||
- Retry logic and fallback behavior
|
||||
- Existing DB data is never deleted on failure
|
||||
"""
|
||||
from datetime import date
|
||||
from json import JSONDecodeError
|
||||
from unittest.mock import patch, MagicMock, call
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from sqlalchemy.pool import StaticPool
|
||||
|
||||
from app.core.database import Base
|
||||
from app.models.stock import ETF, Valuation
|
||||
from app.services.collectors.etf_collector import ETFCollector
|
||||
from app.services.collectors.valuation_collector import ValuationCollector
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db():
|
||||
"""In-memory SQLite database for testing."""
|
||||
engine = create_engine(
|
||||
"sqlite:///:memory:",
|
||||
connect_args={"check_same_thread": False},
|
||||
poolclass=StaticPool,
|
||||
)
|
||||
Base.metadata.create_all(bind=engine)
|
||||
Session = sessionmaker(bind=engine)
|
||||
session = Session()
|
||||
yield session
|
||||
session.close()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
# ── ETFCollector Tests ──
|
||||
|
||||
|
||||
class TestETFCollectorResilience:
|
||||
|
||||
@patch("app.services.collectors.etf_collector.time.sleep")
|
||||
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
|
||||
def test_json_decode_error_retries_once(self, mock_etf_cls, mock_sleep, db):
|
||||
"""JSONDecodeError on first attempt triggers 1 retry with 3s delay."""
|
||||
mock_fetcher = MagicMock()
|
||||
mock_fetcher.fetch.side_effect = [
|
||||
JSONDecodeError("msg", "doc", 0),
|
||||
pd.DataFrame(), # retry returns empty
|
||||
]
|
||||
mock_etf_cls.return_value = mock_fetcher
|
||||
|
||||
collector = ETFCollector(db)
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 0
|
||||
assert mock_fetcher.fetch.call_count == 2
|
||||
mock_sleep.assert_called_once_with(3)
|
||||
|
||||
@patch("app.services.collectors.etf_collector.time.sleep")
|
||||
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
|
||||
def test_connection_error_retries_and_returns_zero(self, mock_etf_cls, mock_sleep, db):
|
||||
"""ConnectionError on both attempts returns 0 without raising."""
|
||||
mock_fetcher = MagicMock()
|
||||
mock_fetcher.fetch.side_effect = ConnectionError("timeout")
|
||||
mock_etf_cls.return_value = mock_fetcher
|
||||
|
||||
collector = ETFCollector(db)
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 0
|
||||
assert mock_fetcher.fetch.call_count == 2
|
||||
|
||||
@patch("app.services.collectors.etf_collector.time.sleep")
|
||||
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
|
||||
def test_retry_succeeds_on_second_attempt(self, mock_etf_cls, mock_sleep, db):
|
||||
"""If first attempt fails but retry succeeds, data is processed normally."""
|
||||
good_df = pd.DataFrame([{
|
||||
"ISU_SRT_CD": "069500",
|
||||
"ISU_ABBRV": "KODEX 200",
|
||||
"IDX_ASST_CLSS_NM": "주식",
|
||||
"IDX_MKT_CLSS_NM": "코스피",
|
||||
"ETF_TOT_FEE": "0.15",
|
||||
}])
|
||||
mock_fetcher = MagicMock()
|
||||
mock_fetcher.fetch.side_effect = [
|
||||
JSONDecodeError("msg", "doc", 0),
|
||||
good_df,
|
||||
]
|
||||
mock_etf_cls.return_value = mock_fetcher
|
||||
|
||||
collector = ETFCollector(db)
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 1
|
||||
|
||||
@patch("app.services.collectors.etf_collector.time.sleep")
|
||||
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
|
||||
def test_failure_does_not_delete_existing_data(self, mock_etf_cls, mock_sleep, db):
|
||||
"""Existing ETF records are preserved when fetch fails."""
|
||||
db.execute(
|
||||
ETF.__table__.insert().values(
|
||||
ticker="069500", name="KODEX 200",
|
||||
asset_class="equity", market="코스피",
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
mock_fetcher = MagicMock()
|
||||
mock_fetcher.fetch.side_effect = JSONDecodeError("msg", "doc", 0)
|
||||
mock_etf_cls.return_value = mock_fetcher
|
||||
|
||||
collector = ETFCollector(db)
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 0
|
||||
existing = db.query(ETF).filter_by(ticker="069500").first()
|
||||
assert existing is not None
|
||||
assert existing.name == "KODEX 200"
|
||||
|
||||
|
||||
# ── ValuationCollector Tests ──
|
||||
|
||||
|
||||
class TestValuationCollectorResilience:
|
||||
|
||||
@patch("app.services.collectors.valuation_collector.pykrx_stock")
|
||||
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
|
||||
def test_key_error_falls_back_to_previous_days(self, mock_biz, mock_pykrx, db):
|
||||
"""KeyError triggers fallback to previous business days."""
|
||||
good_df = pd.DataFrame(
|
||||
{"PER": [10.0], "PBR": [1.5], "DIV": [2.0]},
|
||||
index=["005930"],
|
||||
)
|
||||
mock_pykrx.get_market_fundamental_by_ticker.side_effect = [
|
||||
KeyError("BPS"), # day 0
|
||||
KeyError("PER"), # day -1
|
||||
good_df, # day -2 succeeds
|
||||
]
|
||||
|
||||
collector = ValuationCollector(db, biz_day="20260327")
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 1
|
||||
assert mock_pykrx.get_market_fundamental_by_ticker.call_count == 3
|
||||
|
||||
@patch("app.services.collectors.valuation_collector.pykrx_stock")
|
||||
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
|
||||
def test_all_days_fail_returns_zero(self, mock_biz, mock_pykrx, db):
|
||||
"""When all 4 date attempts fail, returns 0 without raising."""
|
||||
mock_pykrx.get_market_fundamental_by_ticker.side_effect = KeyError("BPS")
|
||||
|
||||
collector = ValuationCollector(db, biz_day="20260327")
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 0
|
||||
assert mock_pykrx.get_market_fundamental_by_ticker.call_count == 4
|
||||
|
||||
@patch("app.services.collectors.valuation_collector.pykrx_stock")
|
||||
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
|
||||
def test_json_decode_error_handled(self, mock_biz, mock_pykrx, db):
|
||||
"""JSONDecodeError is caught and triggers date fallback."""
|
||||
mock_pykrx.get_market_fundamental_by_ticker.side_effect = JSONDecodeError("msg", "doc", 0)
|
||||
|
||||
collector = ValuationCollector(db, biz_day="20260327")
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 0
|
||||
|
||||
@patch("app.services.collectors.valuation_collector.pykrx_stock")
|
||||
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
|
||||
def test_empty_df_triggers_fallback(self, mock_biz, mock_pykrx, db):
|
||||
"""Empty DataFrame (not exception) also triggers fallback."""
|
||||
good_df = pd.DataFrame(
|
||||
{"PER": [15.0], "PBR": [2.0], "DIV": [1.5]},
|
||||
index=["005930"],
|
||||
)
|
||||
mock_pykrx.get_market_fundamental_by_ticker.side_effect = [
|
||||
pd.DataFrame(), # empty
|
||||
good_df, # day -1 has data
|
||||
]
|
||||
|
||||
collector = ValuationCollector(db, biz_day="20260327")
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 1
|
||||
|
||||
@patch("app.services.collectors.valuation_collector.pykrx_stock")
|
||||
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
|
||||
def test_failure_preserves_existing_data(self, mock_biz, mock_pykrx, db):
|
||||
"""Existing valuation records are preserved when fetch fails."""
|
||||
db.execute(
|
||||
Valuation.__table__.insert().values(
|
||||
ticker="005930", base_date=date(2026, 3, 26),
|
||||
per=10.0, pbr=1.5, dividend_yield=2.0,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
mock_pykrx.get_market_fundamental_by_ticker.side_effect = KeyError("BPS")
|
||||
|
||||
collector = ValuationCollector(db, biz_day="20260327")
|
||||
result = collector.collect()
|
||||
|
||||
assert result == 0
|
||||
existing = db.query(Valuation).filter_by(ticker="005930").first()
|
||||
assert existing is not None
|
||||
assert existing.per == 10.0
|
||||
Loading…
x
Reference in New Issue
Block a user