fix: add resilience to ETFCollector and ValuationCollector
Some checks failed
Deploy to Production / deploy (push) Has been cancelled

- ETFCollector: retry once on JSONDecodeError/ConnectionError with 3s delay
- ValuationCollector: fallback to previous 3 business days on KeyError/empty data
- Both: graceful skip on persistent failure, never delete existing DB data
- Add test_collector_resilience.py (9 tests)
This commit is contained in:
머니페니 2026-03-29 22:45:20 +09:00
parent 3dcbcd3080
commit 862c1637bd
9 changed files with 381 additions and 5 deletions

View File

@ -0,0 +1,4 @@
{
"version": 1,
"setupCompletedAt": "2026-03-28T14:03:58.550Z"
}

7
HEARTBEAT.md Normal file
View File

@ -0,0 +1,7 @@
# HEARTBEAT.md Template
```markdown
# Keep this file empty (or with only comments) to skip heartbeat API calls.
# Add tasks below when you want the agent to check something periodically.
```

23
IDENTITY.md Normal file
View File

@ -0,0 +1,23 @@
# IDENTITY.md - Who Am I?
_Fill this in during your first conversation. Make it yours._
- **Name:**
_(pick something you like)_
- **Creature:**
_(AI? robot? familiar? ghost in the machine? something weirder?)_
- **Vibe:**
_(how do you come across? sharp? warm? chaotic? calm?)_
- **Emoji:**
_(your signature — pick one that feels right)_
- **Avatar:**
_(workspace-relative path, http(s) URL, or data URI)_
---
This isn't just metadata. It's the start of figuring out who you are.
Notes:
- Save this file at the workspace root as `IDENTITY.md`.
- For avatars, use a workspace-relative path like `avatars/openclaw.png`.

38
SOUL.md Normal file
View File

@ -0,0 +1,38 @@
# SOUL.md - 갤포
## 정체성
- **이름:** 갤포 (GalPo)
- **역할:** galaxis-po 전담 개발/운영 에이전트
- **이모지:** 🛠️
- **성격:** 신뢰감 있는 시니어 개발자. 간결하고 정확한 커뮤니케이션. 불필요한 수식어 없이 핵심만 전달.
## 전문 분야
1. **코딩** — FastAPI(Python) + Next.js(TypeScript) 풀스택 개발
2. **E2E 테스트** — Playwright 기반 테스트 작성 및 실행
3. **퀀트 전략 리서치** — 김종봉 전략 기반 백테스팅, 신호 생성, 포트폴리오 관리
## 작업 원칙
- `docs/plans/` 설계 문서를 먼저 확인한 후 코딩
- `quant.md` 전략 로직 임의 변경 금지
- 테스트 없는 비즈니스 로직 추가 금지
- `.env`, `docker-compose.prod.yml` 수정 금지
- 작업 완료 시 반드시 보고 형식 준수
## 보고 형식
```
완료: [작업명]
변경 파일: [파일 목록]
주요 내용: [한 줄 요약]
주의사항: [있을 경우만]
```
## 톤
- 한국어 사용
- 군더더기 없는 개발자 톤
- "했습니다" 보다 "완료", "확인됨" 스타일
- 문제 발견 시 즉시 보고, 해결책 같이 제시
- **모든 답변 첫 줄에 `🛠️ [갤포]` 태그를 붙인다** (에이전트 식별용)
## 상위 보고
- 머니페니(메인 에이전트)에게 중요 사항 보고
- 마스터(김현섭)의 직접 지시도 수행

40
TOOLS.md Normal file
View File

@ -0,0 +1,40 @@
# TOOLS.md - Local Notes
Skills define _how_ tools work. This file is for _your_ specifics — the stuff that's unique to your setup.
## What Goes Here
Things like:
- Camera names and locations
- SSH hosts and aliases
- Preferred voices for TTS
- Speaker/room names
- Device nicknames
- Anything environment-specific
## Examples
```markdown
### Cameras
- living-room → Main area, 180° wide angle
- front-door → Entrance, motion-triggered
### SSH
- home-server → 192.168.1.100, user: admin
### TTS
- Preferred voice: "Nova" (warm, slightly British)
- Default speaker: Kitchen HomePod
```
## Why Separate?
Skills are shared. Your setup is yours. Keeping them apart means you can update skills without losing your notes, and share skills without leaking your infrastructure.
---
Add whatever helps you do your job. This is your cheat sheet.

17
USER.md Normal file
View File

@ -0,0 +1,17 @@
# USER.md - About Your Human
_Learn about the person you're helping. Update this as you go._
- **Name:**
- **What to call them:**
- **Pronouns:** _(optional)_
- **Timezone:**
- **Notes:**
## Context
_(What do they care about? What projects are they working on? What annoys them? What makes them laugh? Build this over time.)_
---
The more you know, the better you can help. But remember — you're learning about a person, not building a dossier. Respect the difference.

View File

@ -2,6 +2,8 @@
ETF master data collector from KRX.
"""
import logging
import time
from json import JSONDecodeError
import pandas as pd
from pykrx.website.krx.etx.core import ETF_전종목기본종목
@ -40,9 +42,23 @@ class ETFCollector(BaseCollector):
else:
return AssetClass.MIXED.value
def _fetch_etf_data(self) -> pd.DataFrame:
"""Fetch ETF data with 1 retry on failure."""
last_exc = None
for attempt in range(2):
try:
return ETF_전종목기본종목().fetch()
except (JSONDecodeError, ConnectionError, ValueError, KeyError) as e:
last_exc = e
if attempt == 0:
logger.warning(f"ETF fetch failed (attempt 1/2), retrying in 3s: {e}")
time.sleep(3)
logger.error(f"ETF fetch failed after 2 attempts: {last_exc}")
return pd.DataFrame()
def collect(self) -> int:
"""Collect ETF master data."""
df = ETF_전종목기본종목().fetch()
df = self._fetch_etf_data()
if df.empty:
logger.warning("No ETF data returned from KRX.")

View File

@ -2,7 +2,8 @@
Valuation data collector using pykrx.
"""
import logging
from datetime import datetime
from datetime import datetime, timedelta
from json import JSONDecodeError
import pandas as pd
from pykrx import stock as pykrx_stock
@ -41,17 +42,35 @@ class ValuationCollector(BaseCollector):
except (ValueError, TypeError):
return None
def _fetch_fundamental_data(self) -> tuple[pd.DataFrame, str]:
"""Fetch fundamental data with fallback to previous business days (up to 3 days back)."""
target_date = datetime.strptime(self.biz_day, "%Y%m%d")
for day_offset in range(4): # today + 3 days back
try_date = target_date - timedelta(days=day_offset)
try_date_str = try_date.strftime("%Y%m%d")
try:
df = pykrx_stock.get_market_fundamental_by_ticker(try_date_str, market="ALL")
if not df.empty:
if day_offset > 0:
logger.info(f"Fell back to {try_date_str} (offset -{day_offset}d)")
return df, try_date_str
except (KeyError, JSONDecodeError, ConnectionError, ValueError) as e:
logger.warning(f"Fundamental fetch failed for {try_date_str}: {e}")
continue
logger.error(f"Fundamental fetch failed for {self.biz_day} and 3 previous days")
return pd.DataFrame(), self.biz_day
def collect(self) -> int:
"""Collect valuation data."""
fund_df = pykrx_stock.get_market_fundamental_by_ticker(self.biz_day, market="ALL")
fund_df, effective_biz_day = self._fetch_fundamental_data()
if fund_df.empty:
logger.warning(f"No fundamental data returned for {self.biz_day}")
return 0
base_date = datetime.strptime(self.biz_day, "%Y%m%d").date()
base_date = datetime.strptime(effective_biz_day, "%Y%m%d").date()
logger.info(f"Processing {len(fund_df)} valuation records for {self.biz_day}")
logger.info(f"Processing {len(fund_df)} valuation records for {effective_biz_day}")
records = []
for ticker in fund_df.index:

View File

@ -0,0 +1,212 @@
"""
Unit tests for ETFCollector and ValuationCollector resilience.
Tests verify that collectors handle KRX API failures gracefully:
- JSONDecodeError, ConnectionError, KeyError from pykrx
- Retry logic and fallback behavior
- Existing DB data is never deleted on failure
"""
from datetime import date
from json import JSONDecodeError
from unittest.mock import patch, MagicMock, call
import pandas as pd
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import StaticPool
from app.core.database import Base
from app.models.stock import ETF, Valuation
from app.services.collectors.etf_collector import ETFCollector
from app.services.collectors.valuation_collector import ValuationCollector
@pytest.fixture
def db():
"""In-memory SQLite database for testing."""
engine = create_engine(
"sqlite:///:memory:",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
Base.metadata.create_all(bind=engine)
Session = sessionmaker(bind=engine)
session = Session()
yield session
session.close()
Base.metadata.drop_all(bind=engine)
# ── ETFCollector Tests ──
class TestETFCollectorResilience:
@patch("app.services.collectors.etf_collector.time.sleep")
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
def test_json_decode_error_retries_once(self, mock_etf_cls, mock_sleep, db):
"""JSONDecodeError on first attempt triggers 1 retry with 3s delay."""
mock_fetcher = MagicMock()
mock_fetcher.fetch.side_effect = [
JSONDecodeError("msg", "doc", 0),
pd.DataFrame(), # retry returns empty
]
mock_etf_cls.return_value = mock_fetcher
collector = ETFCollector(db)
result = collector.collect()
assert result == 0
assert mock_fetcher.fetch.call_count == 2
mock_sleep.assert_called_once_with(3)
@patch("app.services.collectors.etf_collector.time.sleep")
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
def test_connection_error_retries_and_returns_zero(self, mock_etf_cls, mock_sleep, db):
"""ConnectionError on both attempts returns 0 without raising."""
mock_fetcher = MagicMock()
mock_fetcher.fetch.side_effect = ConnectionError("timeout")
mock_etf_cls.return_value = mock_fetcher
collector = ETFCollector(db)
result = collector.collect()
assert result == 0
assert mock_fetcher.fetch.call_count == 2
@patch("app.services.collectors.etf_collector.time.sleep")
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
def test_retry_succeeds_on_second_attempt(self, mock_etf_cls, mock_sleep, db):
"""If first attempt fails but retry succeeds, data is processed normally."""
good_df = pd.DataFrame([{
"ISU_SRT_CD": "069500",
"ISU_ABBRV": "KODEX 200",
"IDX_ASST_CLSS_NM": "주식",
"IDX_MKT_CLSS_NM": "코스피",
"ETF_TOT_FEE": "0.15",
}])
mock_fetcher = MagicMock()
mock_fetcher.fetch.side_effect = [
JSONDecodeError("msg", "doc", 0),
good_df,
]
mock_etf_cls.return_value = mock_fetcher
collector = ETFCollector(db)
result = collector.collect()
assert result == 1
@patch("app.services.collectors.etf_collector.time.sleep")
@patch("app.services.collectors.etf_collector.ETF_전종목기본종목")
def test_failure_does_not_delete_existing_data(self, mock_etf_cls, mock_sleep, db):
"""Existing ETF records are preserved when fetch fails."""
db.execute(
ETF.__table__.insert().values(
ticker="069500", name="KODEX 200",
asset_class="equity", market="코스피",
)
)
db.commit()
mock_fetcher = MagicMock()
mock_fetcher.fetch.side_effect = JSONDecodeError("msg", "doc", 0)
mock_etf_cls.return_value = mock_fetcher
collector = ETFCollector(db)
result = collector.collect()
assert result == 0
existing = db.query(ETF).filter_by(ticker="069500").first()
assert existing is not None
assert existing.name == "KODEX 200"
# ── ValuationCollector Tests ──
class TestValuationCollectorResilience:
@patch("app.services.collectors.valuation_collector.pykrx_stock")
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
def test_key_error_falls_back_to_previous_days(self, mock_biz, mock_pykrx, db):
"""KeyError triggers fallback to previous business days."""
good_df = pd.DataFrame(
{"PER": [10.0], "PBR": [1.5], "DIV": [2.0]},
index=["005930"],
)
mock_pykrx.get_market_fundamental_by_ticker.side_effect = [
KeyError("BPS"), # day 0
KeyError("PER"), # day -1
good_df, # day -2 succeeds
]
collector = ValuationCollector(db, biz_day="20260327")
result = collector.collect()
assert result == 1
assert mock_pykrx.get_market_fundamental_by_ticker.call_count == 3
@patch("app.services.collectors.valuation_collector.pykrx_stock")
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
def test_all_days_fail_returns_zero(self, mock_biz, mock_pykrx, db):
"""When all 4 date attempts fail, returns 0 without raising."""
mock_pykrx.get_market_fundamental_by_ticker.side_effect = KeyError("BPS")
collector = ValuationCollector(db, biz_day="20260327")
result = collector.collect()
assert result == 0
assert mock_pykrx.get_market_fundamental_by_ticker.call_count == 4
@patch("app.services.collectors.valuation_collector.pykrx_stock")
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
def test_json_decode_error_handled(self, mock_biz, mock_pykrx, db):
"""JSONDecodeError is caught and triggers date fallback."""
mock_pykrx.get_market_fundamental_by_ticker.side_effect = JSONDecodeError("msg", "doc", 0)
collector = ValuationCollector(db, biz_day="20260327")
result = collector.collect()
assert result == 0
@patch("app.services.collectors.valuation_collector.pykrx_stock")
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
def test_empty_df_triggers_fallback(self, mock_biz, mock_pykrx, db):
"""Empty DataFrame (not exception) also triggers fallback."""
good_df = pd.DataFrame(
{"PER": [15.0], "PBR": [2.0], "DIV": [1.5]},
index=["005930"],
)
mock_pykrx.get_market_fundamental_by_ticker.side_effect = [
pd.DataFrame(), # empty
good_df, # day -1 has data
]
collector = ValuationCollector(db, biz_day="20260327")
result = collector.collect()
assert result == 1
@patch("app.services.collectors.valuation_collector.pykrx_stock")
@patch("app.services.collectors.base.BaseCollector._get_latest_biz_day", return_value="20260327")
def test_failure_preserves_existing_data(self, mock_biz, mock_pykrx, db):
"""Existing valuation records are preserved when fetch fails."""
db.execute(
Valuation.__table__.insert().values(
ticker="005930", base_date=date(2026, 3, 26),
per=10.0, pbr=1.5, dividend_yield=2.0,
)
)
db.commit()
mock_pykrx.get_market_fundamental_by_ticker.side_effect = KeyError("BPS")
collector = ValuationCollector(db, biz_day="20260327")
result = collector.collect()
assert result == 0
existing = db.query(Valuation).filter_by(ticker="005930").first()
assert existing is not None
assert existing.per == 10.0