Fix date parsing, timezone, and type errors in Yahoo collector #2118
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Use pd.to_datetime(utc=True) + tz_convert(None) to handle mixed date formats and timezone-aware/naive inputs in pandas >= 2.0 - Add pd.to_numeric(errors="coerce") before arithmetic on columns that may contain string data from CSV reads - Apply same utc=True fix in base.py Normalize._executor date filtering Fixes microsoft#1981
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
ValueErroron mixed date formats (YYYY-MM-DDvsYYYY-MM-DD HH:MM:SS) by usingpd.to_datetime(utc=True)which handles mixed formats across all pandas versionsAttributeError: 'Index' object has no attribute 'tz_localize'by switching totz_convert(None)afterutc=TrueconversionTypeError: unsupported operand type(s) for /: 'str' and 'float'by addingpd.to_numeric(errors="coerce")before arithmetic operations on columns that may contain string data from CSV readsChanges
scripts/data_collector/yahoo/collector.py: Fixnormalize_yahoodate handling (lines 395-396), add numeric coercion inadjusted_priceand_manual_adj_datascripts/data_collector/base.py: FixNormalize._executordate filtering (line 308)Test plan
fillna(method="ffill")deprecation warning mentioned in the issue is already fixed in the current codebase (.ffill()is used)--skip_downloadare feature requests beyond the scope of this bug fix PRFixes #1981