WIP: Reimplementing `search_dates` by gavishpoddar · Pull Request #945 · scrapinghub/dateparser

gavishpoddar · 2021-07-16T22:30:54Z

Reimplementing and simplifying `search_dates`

A reimplemented and simplified search_dates which more directly uses dateparser.parse, improves accuracy and fixes many bugs

New Feature:

search_first_date - searches and returns the first date from the given text.

NOTE: This PR is inspired by the previous implementation of search_dates and #931.

TODO

gavishpoddar · 2021-07-16T23:03:29Z

This PR will also solve many issues with the `search_dates`.

date has year inside brackets #924
search_dates() wrong result for 12:xx am #894
Errors in processing dates in Russian text #918
search_dates cannot find complete date #521
False positives when searching dates #582
search_dates is... #681
J, K, W, X, Y letters confuse the parser #774
Mistake search_dates for ru language #706 (Correct datetime object but wrong string) (related issue 2)
Parse whole words only #856 (Correct datetime object but wrong string) (related issue 2)

Somewhat address.

Month and Month Year (e.g. April and May 2019) will return None #507 (Now returns date object but not range)
Date ambiguities during search_dates #689 (Now returnes datetime.datetime(2019, 2, 10, 0, 0) but the issue is not clear)
Search dates failing to work due to lack of commas? #695
Dates not getting recognised or wrongly recognised if there are some numbers in the text #787 (Now returnes datetime object but the issue is not clear)

Some other issue fixed:

2021-08-04T14:21:37+05:30

would parse 05:30 only after this PR

2021-08-04T14:21:37 and 05:30

NOTE: THIS LIST IS NO LONGER ACCURATE ANYMORE

gavishpoddar · 2021-07-17T11:58:24Z

Hi, I need a suggestion should I use translated chunks or the original chunks to further parse the data objects. I have currently used translated chunks instead of original chunks as this increased accuracy in some basic tests.

Thanks, and please suggest.

noviluni · 2021-07-19T07:14:00Z

Hi, I need a suggestion should I use translated chunks or the original chunks to further parse the data objects. I have currently used translated chunks instead of original chunks as this increased accuracy in some basic tests.

Thanks, and please suggest.

Hi @gavishpoddar, using translated chunks makes sense. 👍

codecov · 2021-07-19T07:16:08Z

Codecov Report

Base: 98.23% // Head: 98.10% // Decreases project coverage by -0.12% ⚠️

Coverage data is based on head (99e66c6) compared to base (0f5d9c5).
Patch coverage: 96.12% of modified lines in pull request are covered.

❗ Current head 99e66c6 differs from pull request most recent head 2935aae. Consider uploading reports for the commit 2935aae to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #945      +/-   ##
==========================================
- Coverage   98.23%   98.10%   -0.13%     
==========================================
  Files         232      235       +3     
  Lines        2604     2692      +88     
==========================================
+ Hits         2558     2641      +83     
- Misses         46       51       +5

Impacted Files	Coverage Δ
dateparser/search/search.py	`94.73% <94.33%> (-4.64%)`	⬇️
dateparser/search/__init__.py	`100.00% <100.00%> (ø)`
dateparser/search/languages.py	`100.00% <100.00%> (ø)`
dateparser/date_parser.py	`93.75% <0.00%> (-0.19%)`	⬇️
dateparser/date.py	`99.24% <0.00%> (-0.04%)`	⬇️
dateparser/parser.py	`99.01% <0.00%> (-0.01%)`	⬇️
dateparser/utils/__init__.py	`95.27% <0.00%> (ø)`
dateparser/languages/locale.py	`98.69% <0.00%> (ø)`
dateparser/conf.py	`100.00% <0.00%> (ø)`
dateparser/__init__.py	`100.00% <0.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

gavishpoddar · 2021-07-21T18:15:06Z

Hi @noviluni, I have made the changes for tests to work. Can you please approve the workflow approval?

lopuhin · 2021-07-22T06:36:28Z

@gavishpoddar workflows approved 👍

gavishpoddar · 2021-08-03T17:55:57Z

Hi @noviluni, @lopuhin, @kishan3
The new search_dates implementation is nearing its completion.

Code reviews would be great to receive.

gavishpoddar · 2021-08-03T17:59:00Z

I need help with one test

StaticTzInfo 'UTC' is expected.
UTC is received from new search_dates

dateparser/languages/locale.py

gavishpoddar · 2021-08-06T10:40:32Z

Hi after some changes the codes are very much compatible with the old search_dates. Some tests are modified as the working of the search_dates has been changed.

Currently, this PR is left with some docs changes (just replacing the current with new docs and search_first_date).

gavishpoddar · 2021-08-24T14:07:45Z

Replaced previous search_dates with the new implimentaion

lopuhin

Thanks @gavishpoddar , I left some comments regarding tests. Please tell if you need advice with how to implement xfail.

dateparser/search/__init__.py

tests/test_search.py

…rser into search_dates

lopuhin

Thanks @gavishpoddar a few docs suggestions

dateparser/search/__init__.py

Co-authored-by: Konstantin Lopuhin <kostia.lopuhin@gmail.com>

gavishpoddar · 2021-09-07T16:58:25Z

Hey, this PR is updated with the #932

dateparser/search/__init__.py

…rser into search_dates

Eckii24 · 2022-08-06T14:20:12Z

Hi @gavishpoddar,

I found your PR to improve the behavior of search_dates.
There has been no further progress for almost a year. What is the current status of this PR?

Is there anything I can do to help get this PR merged?

Gallaecio · 2022-12-01T10:39:32Z

dateparser/search/search.py

    def __init__(self):
-        self.loader = LocaleDataLoader()
-        self.available_language_map = self.loader.get_locale_map()
-        self.search = _ExactLanguageSearch(self.loader)


The removal of DateSearchWithDetection.search is backward-incompatible.

I can create a shortcut the make DateSearchWithDetection.search and add a deprecation warning or simply rename.

Please suggest a preferred action.

I think keeping the old objects around with their old names, logging a warning when used (i.e. by exposing them through a property whose getter logs a warning), would be ideal.

Implimenting new search_dates

02220da

gavishpoddar mentioned this pull request Jul 16, 2021

Fixing #894 and correcting tests #901

Closed

gavishpoddar changed the title ~~[WIP] Reimplementing search_dates & fixing search translation~~ [WIP] Reimplementing search_dates & fixing search translation Jul 17, 2021

gavishpoddar added 2 commits July 21, 2021 07:22

Fixing DATE_ORDER, implimenting deep_search, tests

f933d3a

Unproving _joint_parse with data_carry accurate_return_text, deep_search

77727b5

gavishpoddar added 2 commits July 22, 2021 20:41

implementing _final_text_clean()

e7f38e8

Simplifying text_clean and modifying tests

962066c

gavishpoddar mentioned this pull request Jul 26, 2021

search_dates : Issues paring dates with period #952

Open

gavishpoddar added 5 commits July 28, 2021 12:15

Implementing relative date

624ac8e

Fixing tests

42ca6f6

secondary_split_implimentation

51749a2

positional args to keyword argument

f5e4635

Micro fixes

121b15f

gavishpoddar added 3 commits August 3, 2021 18:04

Removing codes now part of scrapinghub#953

2cd93f0

adding check_settings

006d2a5

implimenting double_punctuation_split

10404c9

noviluni reviewed Aug 5, 2021

View reviewed changes

dateparser/languages/locale.py Show resolved Hide resolved

gavishpoddar added 2 commits August 6, 2021 09:00

Updating docs and removing test (TMP)

22596e0

cleaning code, adding tests, improving coverage

b799dfb

gavishpoddar requested a review from noviluni August 6, 2021 10:40

gavishpoddar changed the title ~~[WIP] Reimplementing search_dates & fixing search translation~~ Reimplementing search_dates & fixing search translation Aug 6, 2021

adding test

5dabc62

lopuhin reviewed Aug 27, 2021

View reviewed changes

dateparser/search/__init__.py Outdated Show resolved Hide resolved

tests/test_search.py Outdated Show resolved Hide resolved

tests/test_search.py Show resolved Hide resolved

tests/test_search.py Outdated Show resolved Hide resolved

tests/test_search.py Outdated Show resolved Hide resolved

gavishpoddar added 5 commits August 27, 2021 09:52

fixing doc string

ab1778d

fixing doc string

14adf89

Merge branch 'search_dates' of https://github.com/gavishpoddar/datepa…

d57223a

…rser into search_dates

updating xfail

88afa30

updating tests

9209f3d

lopuhin reviewed Sep 1, 2021

View reviewed changes

dateparser/search/__init__.py Outdated Show resolved Hide resolved

dateparser/search/__init__.py Outdated Show resolved Hide resolved

gavishpoddar and others added 4 commits September 1, 2021 15:32

Apply suggestions from code review

85254e0

Co-authored-by: Konstantin Lopuhin <kostia.lopuhin@gmail.com>

Merge branch 'master' into search_dates

e4604e6

Updates

4f119dd

Fixing upstraem merges

e6da4be

lopuhin reviewed Sep 8, 2021

View reviewed changes

dateparser/search/__init__.py Outdated Show resolved Hide resolved

gavishpoddar added 5 commits September 9, 2021 06:29

DateSearch -> DateSearchWithDetection

f6116bf

Merge branch 'scrapinghub:master' into search_dates

0525cdc

updating test with xfail

96b91c0

Merge branch 'search_dates' of https://github.com/gavishpoddar/datepa…

b9d12f3

…rser into search_dates

minor fixes

99e66c6

serhii73 closed this Dec 1, 2022

serhii73 reopened this Dec 1, 2022

serhii73 requested review from Gallaecio, serhii73 and wRAR December 1, 2022 08:47

serhii73 mentioned this pull request Dec 1, 2022

[WIP] add find_date function and tests: finds the first date #931

Open

5 tasks

Gallaecio reviewed Dec 1, 2022

View reviewed changes

gavishpoddar changed the title ~~Reimplementing search_dates~~ WIP: Reimplementing search_dates Dec 7, 2022

Merge branch 'master' into search_dates

2935aae

Conversation

gavishpoddar commented Jul 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reimplementing and simplifying search_dates

TODO

Uh oh!

gavishpoddar commented Jul 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR will also solve many issues with the search_dates.

Somewhat address.

Some other issue fixed:

Uh oh!

gavishpoddar commented Jul 17, 2021

Uh oh!

noviluni commented Jul 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jul 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gavishpoddar commented Jul 21, 2021

Uh oh!

lopuhin commented Jul 22, 2021

Uh oh!

gavishpoddar commented Aug 3, 2021

Uh oh!

gavishpoddar commented Aug 3, 2021

Uh oh!

Uh oh!

gavishpoddar commented Aug 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gavishpoddar commented Aug 24, 2021

Uh oh!

lopuhin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lopuhin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gavishpoddar commented Sep 7, 2021

Uh oh!

Uh oh!

Eckii24 commented Aug 6, 2022

Uh oh!

Gallaecio Dec 1, 2022

Choose a reason for hiding this comment

Uh oh!

gavishpoddar Dec 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gallaecio Dec 1, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

gavishpoddar commented Jul 16, 2021 •

edited

Loading

Reimplementing and simplifying `search_dates`

gavishpoddar commented Jul 16, 2021 •

edited

Loading

This PR will also solve many issues with the `search_dates`.

noviluni commented Jul 19, 2021 •

edited

Loading

codecov bot commented Jul 19, 2021 •

edited

Loading

gavishpoddar commented Aug 6, 2021 •

edited

Loading

gavishpoddar Dec 1, 2022 •

edited

Loading