NY Times fails

[kovidgoyal]

I changed the recipe to download from the internet archive via a caching proxy. It should work much better since it no longer touches the nytimes servers to get articles. However, you may get one day old articles, depending on how frequently the archive's bots crawl nytimes.com

[NSILMike]

I just tried it, and all I got was headlines....

[kovidgoyal]

it takes an hour for the recipe to update.

[NSILMike]

OK I can wait

[mkgtu]

Tried several to download several times time. All failures.
Seemed to be the updated recipe. Wayback machine = true

calibre, version 6.8.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The New York Times

Fetch news from The New York Times
Conversion options changed from defaults:
  output_profile: 'generic_eink_hd'
  verbose: 2
Resolved conversion options
calibre version: 6.8.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_toc_at_end': False,
 'epub_version': '2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x00000232C6D6DA80>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.GenericEinkHD object at 0x00000232C6D6EBC0>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Traceback (most recent call last):
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "site.py", line 82, in <module>
  File "site.py", line 77, in main
  File "site.py", line 49, in run_entry_point
  File "calibre\utils\ipc\worker.py", line 215, in main
  File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
  File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "calibre\ebooks\conversion\plumber.py", line 1108, in run
  File "calibre\customize\conversion.py", line 242, in __call__
  File "calibre\ebooks\conversion\plugins\recipe_input.py", line 138, in convert
  File "calibre\web\feeds\news.py", line 1058, in download
  File "calibre\web\feeds\news.py", line 1227, in build_index
  File "<string>", line 268, in parse_index
  File "<string>", line 137, in parse_todays_page
  File "<string>", line 129, in read_nyt_metadata
  File "<string>", line 126, in read_todays_paper
  File "<string>", line 107, in get_nyt_page
  File "calibre\web\site_parsers\nytimes.py", line 205, in download_url
  File "mechanize\_mechanize.py", line 241, in open_novisit
  File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 404: Not Found

[mkgtu]

Update:
Now the new recipe works. I think. I hope.

I just set it to download automatically every day JUST IN CASE so that maybe tomorrow morning it might get fixed or work again.

As soon as I did that the recipe began to download - as it usually does after setting an auto-download - and the download succeeded completely. We'll see if it works again tomorrow morning.

Also a BTW: The item was emailed to my Kindle with everything left-aligned, which is actually my preference. But I usually have to create custom recipes for all my regular downloads and add the extra css for left alignment because even though when viewing the built in downloads in calibre they display left aligned they don't arrive that way when finally converted by Amazon and delivered to Kindle.
So I was pleasantly surprised when the version from the Internet Archive arrived from Amazon left aligned.

[NSILMike]

It's broken again...

File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 504: Gateway Time-out

[mkgtu]

Erratic.

Two failures today. I success.

First failure was the same as previous poster, a gateway timeout.

The second failure was different.

calibre, version 6.8.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The New York Times

Fetch news from The New York Times
Conversion options changed from defaults:
  output_profile: 'generic_eink_hd'
  verbose: 2
Resolved conversion options
calibre version: 6.8.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_toc_at_end': False,
 'epub_version': '2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001F1F4CADA80>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.GenericEinkHD object at 0x000001F1F4CAEBC0>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Traceback (most recent call last):
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "site.py", line 82, in <module>
  File "site.py", line 77, in main
  File "site.py", line 49, in run_entry_point
  File "calibre\utils\ipc\worker.py", line 215, in main
  File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
  File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "calibre\ebooks\conversion\plumber.py", line 1108, in run
  File "calibre\customize\conversion.py", line 242, in __call__
  File "calibre\ebooks\conversion\plugins\recipe_input.py", line 138, in convert
  File "calibre\web\feeds\news.py", line 1058, in download
  File "calibre\web\feeds\news.py", line 1227, in build_index
  File "<string>", line 268, in parse_index
  File "<string>", line 137, in parse_todays_page
  File "<string>", line 130, in read_nyt_metadata
TypeError: 'NoneType' object is not subscriptable

THIRD TRY WORKED>

[mkgtu]

NO
Third try also failure. It downloaded the full edition. But the edition was TWO DAYS OLD.
Today is Nov 18. The downloaded edition is dated Nov 16 and is indeed the same as the news from two days ago.

This is WAY back.

Sent from my SM-G975U using Tapatalk

[mkgtu]

Tried a fourth time a half hour or more later. Still got the same two day old issue, dated Nov 16. I checked through the articles and they all seemed two days old.
I'm using the built in recipe. No alterations or customizations.

Sent from my SM-G975U using Tapatalk

[mkgtu]

Tried a FIFTH time an hour and a half later and got YESTERDAY'S issue, dated Nov 17. I checked the articles and verified that this is indeed yesterday's NYTimes. Better than two day old edition.

Will try later today to see if I can get today's Nov 18 issue. Meanwhile I've bought today's issue from Amazon (99 cents).

[mkgtu]

I checked the Internet Archive website and found 11 "snapshots". Any of the 10 from 3am forward gave a pretty up to date version of today's edition.

But if you click on the "Today's Paper" link at the top of the page you get a TWO DAY OLD PAPER. I'm not really good at reading these recipes, but it looks to my untrained eye like the current recipe is asking for Today's Paper:


def read_todays_paper(self):
INDEX = 'https://www.nytimes.com/section/todayspaper'
# INDEX = 'file:///t/raw.html'
return self.index_to_soup(self.get_nyt_page(INDEX))

If that's the case, and since it looks like Today's Paper in the Internet Archive is out of date (at least today it is), maybe the recipe shouldn't be asking for it. Might be better off with just the most recent "snapshot" at the time of download.

[mkgtu]

Tried a SIXTH time five and a half hours later (3:30pm Pacific Standard Time) and finally got today's actual Nov 18 paper. So it seems the Internet Archive is very slow to update the "Today's Paper" link on the website to make it point to the current edition - even though the current content is displayed in the current "snapshot" of the website.
Need to make the recipe stop loading the Today's Paper and just get the latest "snapshot". Unless you like to wait till late afternoon to get your early morning paper.

[kovidgoyal]

OK then, in that case I give up. NY times is really not worth the bother to me.

[mkgtu]

This morning, Sat Nov 19 at 6:00am PST, NYT recipe auto-downloaded correctly today's paper. I briefly checked the "Front Page" articles against those on the NYT "Today's Paper" website in my browser and they're a perfect match.

Go figure!

So sometimes the content of "Today's Paper" in the Internet Archive is up to date and other times it's hopelessly behind.

A suggestion, especially since this is a " major" publication: leave both recipes, the regular plus the " way back" versions, in calibre as built in recipes. Maybe add an explanatory note in each about the NYT sometimes blocking the one and the Internet Archive sometimes being slow to update the other. Maybe title this new one "New York Times (archived)" or "New York Times (wayback)" or ... whatever, to distinguish it from the regular version

Then let people pick their poison, one or both.

Including the web version, that would mean three recipes for the NYT. Hopefully one will work when the others don't on any given day.

And for those who would prefer the regular recipe, it would avoid a user trying to amateurishly tweak the "wayback" version's code to restore it to the non-wayback version. Seems like a lot of code was added that's certainly over my head!

Sent from my SM-G975U using Tapatalk