Commit graph

18 commits

Author SHA1 Message Date
Luke Hsiao 661bd9c0fa Set default user agent for external requests
Many servers will return errors (e.g. 400/403) to requests that do not
set a User-Agent header. This results in issues in both the link_checker
and load_data components. With the link_checker these are false positive
dead links. In load_data, remote data fails to be fetched. To mitigate
this issue, this sets a default User-Agent of

    $CARGO_PKG_NAME/$CARGO_PKG_VERSION

Note that the root cause of this regression from zola v0.9.0 is that
reqwest 0.10 changed their default behavior and no longer sets a
User-Agent by default:

    https://github.com/seanmonstar/reqwest/pull/751

Fixes #950.
2020-02-17 17:40:06 -08:00
Vincent Prouillet 15a3ab1a51 One last deps update 2020-02-16 19:17:41 +01:00
Vincent Prouillet a903473a87 Use rustls reqwest feature 2020-02-07 21:16:44 +01:00
djugei 046213ce28 fixed warning (#912) 2020-02-02 17:48:43 -08:00
Sam Ford d7bad732f1 Update reqwest to v0.10 (#892) 2020-02-02 17:48:43 -08:00
Sam Ford 6b5768fd76 Treat 304 as valid, add mock tests, fix mock issue (#900)
* Treat 304 (Not Modified) requests as valid.

* Add tests for 301-to-200 links, 301-to-404 links, and 500 links.
This helps to test redirections and the previously-added
response.status() checking for non-success status codes in check_url().

* Make names for HTTP mock paths unique, to avoid weird behavior. It
seems like mocks with the same path can potentially bleed between
tests, so you may end up with an unexpected response which causes the
test to sometimes pass and sometimes fail.

* Fix Clippy warnings about String::from(format!()).
2020-02-02 17:48:43 -08:00
Sam Ford 11f7a6d114 Mock HTTP requests in tests (#898)
Certain tests involving HTTP requests were sometimes hanging
indefinitely, so this uses Mockito for HTTP mocking. This seemingly
resolves the issue and makes these tests more reliable.

The existing can_fail_404_links test has been renamed to
can_fail_unresolved_links, to represent what actually occurs in the
test. The can_fail_404_links test now deals with a proper 404
response.

Just to be clear, the check_site test in the site component will
still create outgoing HTTP requests (due to the URLs used in the
test_site), so this commit only uses HTTP mocking where possible.
2020-02-02 17:48:43 -08:00
Sam Ford 2f1b592ab4 link_checker: Handle non-success status codes (#897)
The can_fail_404_links() test doesn't encounter a 404 response in
actuality, since the google.comys domain doesn't resolve. When the
test is updated such that the response's status code is a 404, the
test fails because the check_url() function doesn't handle
non-success responses how the test's assertions expect. This commit
updates check_url() to handle non-success responses, treating them
much like errors.
2020-02-02 17:48:43 -08:00
Sam Ford e804f907b2 Use Rust 2018 edition (#885) 2020-02-02 17:48:43 -08:00
Tjeu Kayim 75570d041a Skip link checking for URL with prefix in config (#846) 2020-02-02 17:47:01 -08:00
Tjeu Kayim 6149fd17e1 Skip anchor checking for URL with prefix in config (#812)
* cargo fmt & clippy

* Skip anchor checking for URL with prefix in config
2020-02-02 17:45:16 -08:00
Neil Kistner 52c2b74b39 Add anchor existence checking to link_checker component (#786)
* Add anchor existant checking to link_checker component

* Oops, forgot some changes

* Drop scraper dependency and rework tests

* Handle name attributes
2019-09-04 20:31:19 +02:00
Vincent Prouillet b7ce4e59fb rustfmt 2018-10-31 08:18:57 +01:00
Vincent Prouillet 9dca46cfd3 Correct usage of reqwest headers 2018-09-30 20:20:26 +02:00
Vincent Prouillet f5c88540ed Update reqwest and image 2018-09-30 19:17:51 +02:00
Thomas Hurst 6a5ace62fc link_checker: Set Accept header
As mentioned in #381, crates.io 404's any request without an Accept:
text/html header.  It 200's any request with one, but at least
false-successes don't prevent checking any other links.

This also makes it easier to add a custom User-Agent if desired.

rustfmt and fix a clippy nit (unnecessary return) while I'm here.
2018-08-25 17:17:06 +01:00
Vincent Prouillet 60a52fd2cc Some intellij reformatting 2018-07-31 15:30:49 +02:00
Vincent Prouillet 3e48ff24b5 Add an external link checker
Close #115
2018-07-16 21:15:50 +02:00