* caddyhttp: Explicitly disallow multiple regexp matchers
Fix#5028
Since the matchers would overwrite eachother, we should error out to tell the user their config doesn't make sense.
* Update modules/caddyhttp/matchers.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
This adds:
- `{file.*}` -> `{http.request.uri.path.file.*}`
- `{file_match.*}` -> `{http.matchers.file.*}`
This is a follow-up to #4993 which introduces the new URI file placeholders, and a shortcut for using `file` matcher output.
For example, where the `try_files` directive is a shortcut for this:
```
@try_files file <files...>
rewrite @try_files {http.matchers.file.relative}
```
It could instead be:
```
@try_files file <files...>
rewrite @try_files {file_match.relative}
```
* fileserver: Support glob expansion in file matcher
* Fix tests
* Fix bugs and tests
* Attempt Windows fix, sigh
* debug Windows, WIP
* Continue debugging Windows
* Another attempt at Windows
* Plz Windows
* Cmon...
* Clean up, hope I didn't break anything
* caddyhttp: Support sending HTTP 103 Early Hints
This adds support for early hints in the static_response handler.
* caddyhttp: Don't record 1xx responses
* reverseproxy: Close hijacked conns on reload/quit
We also send a Close control message to both ends of
WebSocket connections. I have tested this many times in
my dev environment with consistent success, although
the variety of scenarios was limited.
* Oops... actually call Close() this time
* CloseMessage --> closeMessage
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Use httpguts, duh
* Use map instead of sync.Map
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* core: Refactor listeners; use SO_REUSEPORT on Unix
Just an experiment for now
* Fix lint by logging error
* TCP Keepalive configuration (#4865)
* initial attempt at TCP Keepalive configuration
* core: implement tcp-keepalive for linux
* move canSetKeepAlive interface
* Godoc for keepalive server parameter
* handle return values
* log keepalive errors
* Clean up after bad merge
* Merge in pluggable network types
From 1edc1a45e3
* Slight refactor, fix from recent merge conflict
Co-authored-by: Karmanyaah Malhotra <karmanyaah.gh@malhotra.cc>
* break up code and use lazy reading and pool bufio.Writer
* close underlying connection when operation failed
* allocate bufWriter and streamWriter only once
* refactor record writing
* rebase from master
* handle err
* Fix type assertion
Also reduce some duplication
* Refactor client and clientCloser for logging
Should reduce allocations
* Minor cosmetic adjustments; apply Apache license
* Appease the linter
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* added the httpError function into the document
* Update templates.go
* Update templates.go
* Fix gofmt
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Until now, the vars matcher has unintentionally lacked parity with the
map directive: the destination placeholders of the map directive would
be expressed as placeholders, i.e. {foo}. But the vars matcher would
not use { }: vars foo value
This looked weird, and was confusing, since it implied that the key
could be dynamic, which doesn't seem helpful here.
I think this is a proper bug fix, since we're not used to accessing
placeholders literally without { } in the Caddyfile.
* core: Plugins can register listener networks
This can be useful for custom listeners.
This feature/API is experimental and may change!
* caddyhttp: Expose server listeners
This allows users to, for example, get upstreams from multiple SRV
endpoints in order (such as primary and secondary clusters).
Also, gofmt went to town on the comments, sigh
Errors returned from the DecisionFunc (whether to get a cert on-demand)
are used as a signal whether to allow a cert or not; *any* error
will forbid cert issuance.
We bubble up the error all the way to the caller, but that caller is the
Go standard library which might gobble it up.
Now we explicitly log connection errors so sysadmins can
ensure their ask endpoints are working.
Thanks to our sponsor AppCove for reporting this!
The intent of "html" is to redirect browser clients only, or those which can evaluate JS and/or meta tags. So return HTTP 200 and no Location header. See #4940.
* fileserver: Support virtual file systems (close#3720)
This change replaces the hard-coded use of os.Open() and os.Stat() with
the use of the new (Go 1.16) io/fs APIs, enabling virtual file systems.
It introduces a new module namespace, caddy.fs, for such file systems.
Also improve documentation for the file server. I realized it was one of
the first modules written for Caddy 2, and the docs hadn't really been
updated since!
* Virtualize FS for file matcher; minor tweaks
* Fix tests and rename dirFS -> osFS
(Since we do not use a root directory, it is dynamic.)
Hahaha this is the ultimate "I have no idea what I'm doing" commit but it
compiles and the tests pass and I declare victory!
... probably broke something, should be tested more.
It is nice that the protobuf dependency becomes indirect now.
Previously, our "duplicate key in server block" logic was flawed because
it did not account for the site's bind address. We defer this check to
when the listener addresses have been assigned, but before we commit
a server block to its listener.
Also refined how network address parsing and joining works, which was
necessary for a less convoluted fix.
* reverseproxy: Implement retry count, alternative to try_duration
* Add Caddyfile support for `retry_match`
* Refactor to deduplicate matcher parsing logic
* Fix lint
See https://caddy.community/t/using-forward-auth-and-writing-my-own-authenticator-in-php/16410, apparently it didn't work when `copy_headers` wasn't used. This is because we were skipping adding a handler to the routes in the "good response handler", but this causes the logic in `reverseproxy.go` to ignore the response handler since it's empty. Instead, we can just always put in the `header` handler, even with an empty `Set` operation, it's just a no-op, but it fixes that condition in the proxy code.
* Make reverse proxy TLS server name replaceable for SNI upstreams.
* Reverted previous TLS server name replacement, and implemented thread safe version.
* Move TLS servername replacement into it's own function
* Moved SNI servername replacement into httptransport.
* Solve issue when dynamic upstreams use wrong protocol upstream.
* Revert previous commit.
Old commit was: Solve issue when dynamic upstreams use wrong protocol upstream.
Id: 3c9806ccb6
* Added SkipTLSPorts option to http transport.
* Fix typo in test config file.
* Rename config option as suggested by Matt
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Update code to match renamed config option.
* Fix typo in config option name.
* Fix another typo that I missed.
* Tests not completing because of apparent wrong ordering of options.
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Make reverse proxy TLS server name replaceable for SNI upstreams.
* Reverted previous TLS server name replacement, and implemented thread safe version.
* Move TLS servername replacement into it's own function
* Moved SNI servername replacement into httptransport.
* Solve issue when dynamic upstreams use wrong protocol upstream.
* Revert previous commit.
Old commit was: Solve issue when dynamic upstreams use wrong protocol upstream.
Id: 3c9806ccb6
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
In v2.5.0, upstream health was fixed such that whether an upstream is
considered healthy or not is mostly up to each individual handler's
config. Since "healthy" is an opinion, it is not a global value.
I unintentionally left in the "healthy" field in the API endpoint for
checking upstreams, and it is now misleading (see #4792).
However, num_requests and fails remains, so health can be determined by
the API client, rather than having it be opaquely (and unhelpfully)
determined for the client.
If we do restore this value later on, it'd need to be replicated once
per reverse_proxy handler according to their individual configs.
* ci: Fix build caching on Windows
I was getting tired of Windows being slow as molasses in our CI jobs, so I went to look at our trusty source of github actions + golang information, and found a somewhat recent commit that actually fixed it. See https://github.com/mvdan/github-actions-golang/commit/4b754729baa709da219a5889c459010d4eda1888
I'll do a 2nd empty commit to re-trigger CI shortly to confirm that it actually fixes it.
* Retrigger CI
* reverseproxy: Improve hashing LB policies with HRW
Previously, if a list of upstreams changed, hash-based LB policies
would be greatly affected because the hash relied on the position of
upstreams in the pool. Highest Random Weight or "rendezvous" hashing
is apparently robust to pool changes. It runs in O(n) instead of
O(log n), but n is very small usually.
* Fix bug and update tests
* httpcaddyfile: Add `{vars.*}` placeholder shortcut
I'm yoinking this from my https://github.com/caddyserver/caddy/pull/4657 PR because I think we should get this in ASAP for v2.5.0 along with the new `vars` directive.
* Sort vars by matchers in reverse
* ci: Ensure we always check for latest version of Go
* Try to force 1.18.1, 1.17.9
* Use includes for the actual go semver
* Use `~` for semver here, apparently
* Try to make tests still run on 1.18.0 for Mac, for now
* caddypki: Load intermediate for signing on-the-fly
Fixes#4517
Big thanks to @maraino for adding an API in `smallstep/certificates` so that we can fix this
* Debug log
* Trying a hunch, does it need to be a pointer receiver?
* Clarify pointer receiver
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* reverseproxy: Sync up `handleUpgradeResponse` with stdlib
I had left this as a TODO for when we bump to minimum 1.17, but I should've realized it was under `internal` so it couldn't be used directly.
Copied the functions we needed for parity. Hopefully this is ok!
* Add tests and fix godoc comments
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
Includes several breaking changes; code base updated accordingly.
- Added lots of context arguments
- Use fs.ErrNotExist
- Rename ACMEManager -> ACMEIssuer; CertificateManager -> Manager
Guh, this is complicated.
Fixes#4640
This also follows up on #4398 (reverting it) which made a change that technically worked, but was incorrect. It changed the condition in `hostsFromKeysNotHTTP` from `&&` to `||`, but then the function no longer did what its name said it would do, and it would return hosts even if they were marked with `http://`, if they used a non-HTTP port. That wasn't the intent of it. The test added in there was kept though, because it is a valid usecase.
The actual fix is to check _earlier_ whether all the addresses explicitly have `http://`, and if so we can short circuit and skip considering the rest.
Lots of the files were using CRLF instead of LF. Mostly my fault cause sometimes I make the files on Windows and VSCode for some reason kept making them with the wrong line endings. Sigh.
Since .txt files typically default to spaces for indentation, I'm also adding an .editorconfig to ensure they use tabs instead
* caddyfile: Support for raw token values, improve `map`, `expression`
* Applied code review comments
* Rename RawVal to ValRaw
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* reverseproxy: New `copy_response` handler for `handle_response` routes
Followup to #4298 and #4388.
This adds a new `copy_response` handler which may only be used in `reverse_proxy`'s `handle_response` routes, which can be used to actually copy the proxy response downstream.
Previously, if `handle_response` was used (with routes, not the status code mode), it was impossible to use the upstream's response body at all, because we would always close the body, expecting the routes to write a new body from scratch.
To implement this, I had to refactor `h.reverseProxy()` to move all the code that came after the `HandleResponse` loop into a new function. This new function `h.finalizeResponse()` takes care of preparing the response by removing extra headers, dealing with trailers, then copying the headers and body downstream.
Since basically what we want `copy_response` to do is invoke `h.finalizeResponse()` at a configurable point in time, we need to pass down the proxy handler, the response, and some other state via a new `req.WithContext(ctx)`. Wrapping a new context is pretty much the only way we have to jump a few layers in the HTTP middleware chain and let a handler pick up this information. Feels a bit dirty, but it works.
Also fixed a bug with the `http.reverse_proxy.upstream.duration` placeholder, it always had the same duration as `http.reverse_proxy.upstream.latency`, but the former was meant to be the time taken for the roundtrip _plus_ copying/writing the response.
* Delete the "Content-Length" header if we aren't copying
Fixes a bug where the Content-Length will mismatch the actual bytes written if we skipped copying the response, so we get a message like this when using curl:
```
curl: (18) transfer closed with 18 bytes remaining to read
```
To replicate:
```
{
admin off
debug
}
:8881 {
reverse_proxy 127.0.0.1:8882 {
@200 status 200
handle_response @200 {
header Foo bar
}
}
}
:8882 {
header Content-Type application/json
respond `{"hello": "world"}` 200
}
```
* Implement `copy_response_headers`, with include/exclude list support
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* opentelemetry: create a new module
* fix imports
* fix test
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddyhttp/opentelemetry/tracer.go
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* rename error ErrUnsupportedTracesProtocol
* replace spaces with tabs in the test data
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Update modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* replace spaces with tabs in the README.md
* use default values for a propagation and exporter protocol
* set http attributes with helper
* simplify code
* Cleanup modules/caddyhttp/opentelemetry/README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update link in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update documentation in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update link to naming spec in README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename module from opentelemetry to tracing
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename span_name to span
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Rename span_name to span
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Simplify otel resource creation
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* handle extra attributes
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.opentelemetry.io/otel/semconv to 1.7.0
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.opentelemetry.io/otel version
* remove environment variable handling
* always use tracecontext,baggage as propagators
* extract tracer name into variable
* rename OpenTelemetry to Tracing
* simplify resource creation
* update go.mod
* rename package from opentelemetry to tracing
* cleanup tests
* update Caddyfile example in README.md
* update README.md
* fix test
* fix module name in README.md
* fix module name in README.md
* change names in README.md and tests
* order imports
* remove redundant tests
* Update documentation README.md
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Fix grammar
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update comments
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* Update comments
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* update go.sum
* update go.sum
* Add otelhttp instrumentation, update OpenTelemetry libraries.
* Use otelhttp instrumentation for instrumenting HTTP requests.
This change uses context.WithValue to inject the next handler into the
request context via a "nextCall" carrier struct, and pass it on to a
standard Go HTTP handler returned by otelhttp.NewHandler. The
underlying handler will extract the next handler from the context,
call it and pass the returned error to the carrier struct.
* use zap.Error() for the error log
* remove README.md
* update dependencies
* clean up the code
* change comment
* move serveHTTP method from separate file
* add syntax to the UnmarshalCaddyfile comment
* go import the file
* admin: Write proper status on invalid requests (#4569) (fix#4561)
* update dependencies
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Vibhav Pant <vibhavp@gmail.com>
Co-authored-by: Alok Naushad <alokme123@gmail.com>
Co-authored-by: Cedric Ziel <cedric@cedric-ziel.com>
* Add a override_domain option to allow DNS chanllenge delegation
CNAME can be used to delegate answering the chanllenge to another DNS
zone. One usage is to reduce the exposure of the DNS credential [1].
Based on the discussion in caddy/certmagic#160, we are adding an option
to allow the user explicitly specify the domain to delegate, instead of
following the CNAME chain.
This needs caddy/certmagic#160.
* rename override_domain to dns_challenge_override_domain
* Update CertMagic; fix spelling
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* reverseproxy: Begin refactor to enable dynamic upstreams
Streamed here: https://www.youtube.com/watch?v=hj7yzXb11jU
* Implement SRV and A/AAA upstream sources
Also get upstreams at every retry loop iteration instead of just once
before the loop. See #4442.
* Minor tweaks from review
* Limit size of upstreams caches
* Add doc notes deprecating LookupSRV
* Provision dynamic upstreams
Still WIP, preparing to preserve health checker functionality
* Rejigger health checks
Move active health check results into handler-specific Upstreams.
Improve documentation regarding health checks and upstreams.
* Deprecation notice
* Add Caddyfile support, use `caddy.Duration`
* Interface guards
* Implement custom resolvers, add resolvers to http transport Caddyfile
* SRV: fix Caddyfile `name` inline arg, remove proto condition
* Use pointer receiver
* Add debug logs
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* reverseproxy: Make shallow-ish clone of the request
* Refactor request cloning into separate function
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
If .ts.net domains are explicitly added to config,
don't try to manage a cert for them (it will fail, and our
implicit Tailscale module will
get those certs at run-time).
Remove /pki/certificates/<ca> endpoint and split into two endpoints:
- GET /pki/ca/<id> to get CA info and certs in JSON format
- GET /pki/ca/<id>/certificates to get cert in PEM chain
* admin: Implement /pki/certificates/<id> API
* pki: Lower "skip_install_trust" log level to INFO
See https://github.com/caddyserver/caddy/issues/4058#issuecomment-976132935
It's not necessary to warn about this, because this was an option explicitly configured by the user. Still useful to log, but we don't need to be so loud about it.
* cmd: Export functions needed for PKI app, return API response to caller
* pki: Rewrite `caddy trust` command to use new admin endpoint instead
* pki: Rewrite `caddy untrust` command to support using admin endpoint
* Refactor cmd and pki packages for determining admin API endpoint
* Update matchers.go
* Update matchers.go
* implementation of zone_id handling
* last changes in zone handling
* give return true values instead of bool
* Apply suggestions from code review
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* changes as suggested
* Apply suggestions from code review
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Update matchers.go
* shortened the Match function
* changed mazcher handling
* Update matchers.go
* delete space
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Huge thank-you to Tailscale (https://tailscale.com) for making this change possible!
This is a great feature for Caddy and Tailscale is a great fit for a standard implementation.
* caddytls: GetCertificate modules; Tailscale
* Caddyfile support for get_certificate
Also fix AP provisioning in case of empty subject list (persist loaded
module on struct, much like Issuers, to surive reprovisioning).
And implement start of HTTP cert getter, still WIP.
* Update modules/caddytls/automation.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Use tsclient package, check status for name
* Implement HTTP cert getter
And use reuse CertMagic's PEM functions for private keys.
* Remove cache option from Tailscale getter
Tailscale does its own caching and we don't need the added complexity...
for now, at least.
* Several updates
- Option to disable cert automation in auto HTTPS
- Support multiple cert managers
- Remove cache feature from cert manager modules
- Minor improvements to auto HTTPS logging
* Run go mod tidy
* Try to get certificates from Tailscale implicitly
Only for domains ending in .ts.net.
I think this is really cool!
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
The TestFileListing test in tplcontext_test has one test that verifies
if directory traversal is not happening. The context root is set to
'/tmp' and then it tries to open '../../../../../etc', which gets
normalized to '/tmp/etc'.
The test then expects an error to be returned, assuming that '/tmp/etc'
does not exist on the system. When it does exist, it results in a test
failure:
```
--- FAIL: TestFileListing (0.00s)
tplcontext_test.go:422: Test 4: Expected error but had none
FAIL
FAIL
github.com/caddyserver/caddy/v2/modules/caddyhttp/templates 0.042s
```
Instead of using '/tmp' as root, use a dedicated directory created with
`os.MkdirTemp()` instead. That way, we know that the directory is empty.
* explain cryptic unix socket listener error related to process kill
https://github.com/caddyserver/caddy/pull/4533
* less ambiguous wording: clean up -> delete
* shorten error message explanation
* link back to pull request in comment for later archeaology
The `net.JoinHostPort()` function has some naiive logic for handling IPv6, it just checks if the host part has a `:` and if so it wraps the host part with `[ ]` but this causes our network type prefix to get wrapped as well, which is invalid for `caddy.NetworkAddress`. Instead, we can just concatenate the host and port manually here to avoid this side-effect.
When this listener code was first written, UsagePool didn't exist. We
can simplify much of the wrapped listener logic by utilizing UsagePool.
This also fixes a bug where new servers were able to clear deadlines
set by old servers, even if the old server didn't get booted out of its
Accept() call yet. And with the deadline cleared, they never would.
(Sometimes. Based on reports and difficulty of reproducing the bug,
this behavior was extremely rare.) I don't know why that happened
exactly, maybe some polling mechanism in the kernel and if the timings
worked out just wrong it would expose the bug.
Anyway, now we ensure that only the closer that set the deadline is the
same one that clears it, ensuring that old servers always return out of
Accept(), because the deadline doesn't get cleared until they do.
Of course, all this hinges on the hope that my suspicions in the middle
of the night are correct and that kernels work the way I think they do
in my head.
Also minor enhancement to UsagePool where if a value errors upon
construction (a very real possibility with listeners), it is removed from
the pool. Not 100% sure the sync logic is correct there, or maybe we
don't have to even put it in the pool until after construction, but it's
subtle either way and I think this is safe... right?
This is a followup to #4407, in response to a report on the forums: https://caddy.community/t/php-fastcgi-phishing-redirection/14542
Turns out that doing `TrimRight` to remove trailing dots, _before_ cleaning the path, will cause double-dots at the end of the path to not be cleaned away as they should. We should instead remove the dots _after_ cleaning.
* caddyhttp: Enhance vars matcher
Enable "or" logic for multiple values.
Fall back to checking placeholders if not a var name.
* Fix tests (thanks @mohammed90 !)
* fastcgi: Fix a TODO, prevent zap using reflection for logging env
* Update modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* reverseproxy: Adjust defaults, document defaults
Related to some of the issues in https://github.com/caddyserver/caddy/issues/4245, a complaint about the proxy transport defaults not being properly documented in https://caddy.community/t/default-values-for-directives/14254/6.
- Dug into the stdlib to find the actual defaults for some of the timeouts and buffer limits, documenting them in godoc so the JSON docs get them next release.
- Moved the keep-alive and dial-timeout defaults from `reverseproxy.go` to `httptransport.go`. It doesn't make sense to set defaults in the proxy, because then any time the transport is configured with non-defaults, the keep-alive and dial-timeout defaults are lost!
- Sped up the dial timeout from 10s to 3s, in practice it rarely makes sense to wait a whole 10s for dialing. A shorter timeout helps a lot with the load balancer retries, so using something lower helps with user experience.
* reverseproxy: Make keepalive interval configurable via Caddyfile
* fastcgi: DialTimeout default for fastcgi transport too
Fixes#4428
It's best to still log handler errors at debug level so that they're hidden by default, but still accessible if additional details are necessary.
This makes it easier for users to find the default browse template if they
want to create a custom template based on that. It also makes it easier to
view the template with proper syntax highlighting.
* caddycmd: Add `--skip-cleanup` to upgrade commands
This is a partial fix for https://github.com/caddyserver/caddy/issues/4057, making it possible to retain the old build of Caddy, in case something went wrong.
* caddycmd: Fix duplicate error message
The error message "download succeeded, but unable to execute" was repeated, because it was both in the `listModules`/`showVersion` functions and in the calling `upgradeBuild` function. Oversight when this was refactored.
* caddycmd: Implement fix for performing cleanup on Windows
Without this, the cleanup operation would fail with an error message like this:
upgrade: download succeeded, but unable to clean up backup binary: remove C:\caddy\caddy.exe.tmp: Access is denied.
* caddycmd: Rename to `--keep-backup`, simplify build constraints
* caddyhttp: Sanitize scheme and host on incoming requests
* reverseproxy: Sanitize the URL scheme and host before proxying
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Ideally this needs to be fixed upstream in github.com/manifoldco/promptui, but it appears unmaintained. Our dependency is extremely indirect:
$ go mod why github.com/juju/ansiterm
# github.com/juju/ansiterm
github.com/caddyserver/caddy/v2/modules/caddypki
github.com/smallstep/certificates/authority
go.step.sm/cli-utils/ui
github.com/manifoldco/promptui
github.com/juju/ansiterm
And it appears that all dependencies in this chain are in conflict with the LGPL license.
Ref:
- https://github.com/manifoldco/promptui/issues/173
- https://github.com/manifoldco/promptui/pull/181
/cc @maraino
The upgrade of smallstep/certificates fixes#4251. The upgrade of CertMagic fixes an issue reported in the forum that a longer timeout was confirmed to resolve (without any particular explanation, but oh well). Other upgrades have minor improvements and seem safe.
* Update tplcontext.go
Add {{ render "/path/to/file.ext" $data }} via funcRender
* Update tplcontext.go
* Refactor funcInclude, add funcImport to enable {{block}} and {{template}}
* Fix funcImport return of nil showing up in html
* Update godocs for and
* Add tests for funcInclude
* Add tests for funcImport
* os.RemoveAll -> os.Remove for TestFuncInclude and TestFuncImport
Related to (closed) Issue #2094 on template inheritance. This PR adds a new function called "import" which works like "include", except it only takes one argument and passes it to the referenced file to be used as "." in that file.
* Update tplcontext.go
Add {{ render "/path/to/file.ext" $data }} via funcRender
* Update tplcontext.go
* Refactor funcInclude, add funcImport to enable {{block}} and {{template}}
* Fix funcImport return of nil showing up in html
* Update godocs for and
* caddyhttp: Add support for triggering errors from `try_files`
* caddyhttp: Use vars instead of placeholders/replacer for matcher errors
* caddyhttp: Add comment for matcher error var key
This generated way too many test jobs, which weren't really that useful. Cross-build is just to keep us posted on which architectures are building okay, so it's not necessary to do it twice. Only plan9 is not working at this point (see https://github.com/caddyserver/caddy/issues/3615)
* encode: ignore flushing until after first write (fix#4314)
The first write will determine if encoding has to be done and will add an Content-Encoding. Until then Flushing has to be delayed so the Content-Encoding header can be added before headers and status code is written. (A passthrough flush would write header and status code)
* Update modules/caddyhttp/encode/encode.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
We realized we made some mistakes with the directive ordering, so we're making some minor adjustments.
`abort` and `error` don't really make sense to be after other handler directives, because you would expect to be able to "fail-fast" and throw an error before falling through to some `file_server` or `respond` typically. So we're moving them up to just before `respond`, i.e. before the common handler directives.
This is also more consistent with our existing examples in the docs, which actually didn't work due to the directive ordering. See https://caddyserver.com/docs/caddyfile/directives/error#examples
Also, `push` doesn't quite make sense to be after `handle`/`route`, since its job is to read from response headers to push additional resources if necessary, and `handle`/`route` may be terminal so push would not be reached if it was declared outside those. And also, it would make sense to be _before_ `templates` because a template _could_ add a `Link` header to the response dynamically.
The commit goreleaser/goreleaser@013bd69126 of GoReleaser is now checking the `go version` prior to executing any of the pre-hooks, which involves setting the current dir of the command to the `build.dir` of the build config. At the time of version check, the buil dir does not exist. It's created in the pre-hook. As a workaround, the build-dir is now created in the Github Action prior to executing goreleaser action.
From reading through the code, I think this code path is now obsoleted by the changes made in https://github.com/caddyserver/caddy/pull/4266.
Basically, `h.flushInterval()` will set the flush interval to `-1` if we're in a bi-directional stream, and the recent PR ensured that `h.copyResponse()` properly flushes headers immediately when the flush interval is non-zero. So now there should be no need to call Flush before calling `h.copyResponse()`.
Some new users mistakenly try to define two sites without braces around each. Doing this can yield a confusing error message saying that their site address is an "unknown directive".
We can do better by keeping track of whether the current site block was parsed with or without a brace, then changing the error message later based on that.
For example, now this invalid config:
```
foo.example.com
respond "foo"
bar.example.com
respond "bar"
```
Will yield this error message:
```
$ caddy adapt
2021/08/22 19:21:31.028 INFO using adjacent Caddyfile
adapt: Caddyfile:4: unrecognized directive: bar.example.com
Did you mean to define a second site? If so, you must use curly braces around each site to separate their configurations.
```
Some users forget to use a comma between their site addresses. This is invalid (commas aren't a valid character in domains) and later parts of the code like certificate automation will try to use this otherwise, which doesn't make sense. Best to error as early as possible.
Example thread on the forums where this happened: https://caddy.community/t/simplify-caddyfile/13281/9
* httpcaddyfile: Add shortcut for proxy hostport placeholder
I've noticed that it's a pretty common pattern to write a proxy like this, when needing to proxy over HTTPS:
```
reverse_proxy https://example.com {
header_up Host {http.reverse_proxy.upstream.hostport}
}
```
I find it pretty hard to remember the exact placeholder to use for this, and I continually need to refer to the docs when I need it. I think a simple fix for this is to add another Caddyfile placeholder for this one to shorten it:
```
reverse_proxy https://example.com {
header_up Host {proxy_hostport}
}
```
* Switch the shortcut name
* adding package command
* add-package command name
* refactoring duplicate code
* fixed by review
* fixed by review
* remove-package command
* commands in different files, common utils
* fix add, remove, upgrade packages in 1 file
* copyright and downloadPath moved
* refactor
* downloadPath do no export
* adding/removing multiple packages
* addPackages/removePackages, comments, command-desc
* add-package, process case len(args) == 0
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* httpcaddyfile: ensure hosts to skip can always be collected
Previously, some hosts that should be skipped in logging would
be missed as the current logic would only collect them after
encountering the first server that would log. This change makes sure
the ServerLogConfig is initialized before iterating over the server
blocks.
* httpcaddyfile: add test case for skip hosts behavior
* feat: implement a simple timer to pull config
mostly referenced to the issue
re #4106
* Update admin.go
use `caddy.Duration`
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Update caddy.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Update admin.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* fix: sync load config when no pull interval provided
try not to make break change
* fix: change PullInterval to LoadInterval
* fix: change pull_interval to load_interval
* Update caddy.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
This commit fixes the `sortByNameDirFirst` variable inside fileserver to
match what browse's default template has.
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
If an email is specified in global options, a site called 'localhost' shouldn't be bunched together with public DNS names in the automation policies, which get the default, public-CA issuers. Fix old test that did this.
I also noticed that these two:
localhost {
}
example.com {
}
and
localhost, example.com {
}
produce slightly different TLS automation policies. The former is what the new test case covers, and we have logic that removes the empty automation policy for localhost so that auto-HTTPS can implicitly create one. (We prefer that whenever possible.) But the latter case produces two automation policies, with the second one being for localhost, with an explicit internal issuer. It's not wrong, just more explicit than it needs to be.
I'd really like to completely rewrite the code from scratch that generates automation policies, hopefully there is a simpler, more correct algorithm.
* Tweak compression settings
zstd: Limit window sizes to 128K to keep memory in control both server and client size.
zstd: Write 0 length frames. This may be needed for compatibility.
zstd: Create fewer encoders. Small memory improvement.
gzip: Allow -2 (Huffman only) and -3 (stateless) compression modes.
* Update modules/caddyhttp/encode/zstd/zstd.go
Update docs.
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
This is the more correct implementation of 23dadc0d86 (#4179)... I think. This commit effectively undoes the revert in 8848df9c5d, but with corrections to the logic.
We *do* need to use the original request path (the path the browser knows) for redirects, since they are external, and rewrites are only internal.
However, if the path was rewritten to a non-canonical path, we should not redirect to canonicalize that, since rewrites are intentional by the site owner. Canonicalizing the path involves modifying only the suffix (base element, or filename) of the path. Thus, if a rewrite involves only the prefix (like how handle_path strips a path prefix), then we can (hopefully!) safely redirect using the original URI since the filename was not rewritten.
So basically, if rewrites modify the filename, we should not canonicalize those requests. If rewrites only modify another part of the path (commonly a prefix), we should be OK to redirect.
Templates are parsed at request-time (like they are in the templates middleware) to allow live changes to the template while the server is running. Fixes race condition.
Also refactored use of a buffer so a buffer put back in the pool will not continue to be used (written to client) in the meantime.
A couple of benchmarks removed due to refactor, which is fine, since we know pooling helps here.
Also split the Caddyfile subdirective keepalive_idle_conns into two properties so the conns and conns_per_host can be set separately.
This is technically a breaking change, but probably anyone who this breaks already had a broken config anyway, and silently fixing it won't help them fix their configs.
While the Caddy project has had very few valid security bug reports over the years, we have a low signal-to-noise ratio with them (lots of invalid reports). Most are out of scope, and it can take too much valuable time for us to determine that. We would prefer researchers do this first. Hopefully these paragraphs spell out much more clearly what we do and don't accept.
* Force auto-renew for OCSP revoked status (maybe) (fix#4191)
* Use latest commit
* go.mod: Use CertMagic v0.14.0 (fix#4191)
Correctly replaces revoked certificates
In the Caddyfile, hosts specified for HTTP sockets (either scheme is "http" or it is on the HTTP port) should not be used as subjects in TLS automation policies (APs).
* logging: Implement dial timeout for net writer (fix#4083)
* Limit how often redials are attempted
This should cause dial blocking to occur only once every 10 seconds at most, but it also means the logger connection might be down for up to 10 seconds after it comes back online; oh well. We shouldn't block for DialTimeout at every single log emission.
* Clarify offline behavior
* caddyfile: Add parse error on site address in `{`
This is an incredibly common mistake made by users, so we should catch it earlier in the parser and give a more friendly message. Often it ends up adapting but with mistakes, or erroring out later due to other site addresses being read as directives.
There's not really ever a situation where a lone '{' is valid at the end of a site address (but I suppose there are edgecases where the user wants to use a path matcher where it ends specifically in `{`, but... why?), so this should be fine.
* Update caddyconfig/caddyfile/parse.go
Turns out this was an oversight, we assumed we could use `{http.response.header.*}` but that doesn't work because those are grabbed from the response writer, and we haven't copied any headers into the response writer yet.
So the fix is to set all the response headers into the replacer at a new namespace before running the handlers.
This adds the `{http.reverse_proxy.header.*}` replacer.
See https://caddy.community/t/empty-http-response-header-x-accel-redirect/12447
We decided that we'll use branches like `2.4` as the target for any changes that we might want to release in a `2.4.x` version like `2.4.1`, so that we can continue to merge changes targeting the next minor release (e.g. `2.5.0`) on master.
Our CI config wasn't set up for this to work properly though, since it was only running checks on PRs targeting master. This should fix it.
I couldn't find a way to do a pattern to only match digits for the branch names from Github's docs, it just looks like a pretty generic glob syntax. But this should do until we get to 3.0
* caddyfile(formatter): fix nesting not decrementing
This is an extremely weird edge-case where if you had a environment variable {}
on one line, a comment on the next line, and the closing of the block on the
following line; the rest of the Caddyfile would be indented further than it
should've been.
ref; https://github.com/matthewpi/vscode-caddyfile-support/issues/13
* run gofmt
* fmt: better way of handling edge case
Followup to #4150, #4151 /cc @ueffel @polarathene
After a bit of discussion with @mholt, we decided to remove `prefer` as a subdirective and just go with using the order implicitly always. Simpler config, simpler docs, etc.
Effectively changes 7776471 and reverts a small part of f35a7fa.
* caddyhttp: Fix fallback for the error handler chain
The fix I went with in the end (after realizing some mistaken assumptions in #4131) is to just make the routes fall back to errorEmptyHandler instead of the non-error empty handler, if Terminal is true, making the routes error-aware. Ultimately this was probably just an oversight when errors was implemented at some point in the early betas of v2.
See https://caddy.community/t/problem-with-basicauth-handle-errors/12243/9 for context.
* Revert "caddyhttp: Fix fallback for the error handler chain"
This reverts commit 95b6ac44a6.
* caddyhttp: Fix via `routes.go`
* fileserver: Fix `file` matcher with empty `try_files`
Fixes https://github.com/caddyserver/caddy/issues/4146
If `TryFiles` is empty, we fill it with `r.URL.Path`. In this case, this is `/`. Then later, in `prepareFilePath()`, we run the replacer (which turns `{path}` into `/` at that point) but `file` remains the original value (and the placeholder is still the placeholder there).
So then `strings.HasSuffix(file, "/")` will be `false` for the placeholder, but `true` for the empty `TryFiles` codepath, because `file` was `/` due to being set to the actual request value beforehand.
This means that `suffix` becomes `//` in that case, so after `sanitizedPathJoin`, it becomes `./`, so `strictFileExists`'s `strings.HasSuffix(file, separator)` codepath will return true.
I think we should change the `m.TryFiles == nil` codepath to `m.TryFiles = []string{"{http.request.uri.path}"}` for consistency. (And maybe consider hoisting this to `Provision` cause there's no point doing this on every request). I don't think this "optimization" of directly using `r.URL.Path` is so valuable, cause it causes this edgecase with directories.
* Update modules/caddyhttp/fileserver/matcher.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* reverseproxy: Add `handle_response` blocks to `reverse_proxy` (#3710)
* reverseproxy: complete handle_response test
* reverseproxy: Change handle_response matchers to use named matchers
reverseproxy: Add support for changing status code
* fastcgi: Remove obsolete TODO
We already have d.Err("transport already specified") in the reverse_proxy parsing code which covers this case
* reverseproxy: Fix support for "4xx" type status codes
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddyhttp: Reorganize response matchers
* reverseproxy: Reintroduce caddyfile.Unmarshaler
* reverseproxy: Add comment mentioning Finalize should be called
Co-authored-by: Maxime Soulé <btik-git@scoubidou.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Below is the report using `benchstat` and cmd:
`go test -run=BenchmarkHeaderREMatcher -bench=BenchmarkHeaderREMatcher -benchmem -count=10`
```
name old time/op new time/op delta
HeaderREMatcher-16 869ns ± 1% 658ns ± 0% -24.29% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
HeaderREMatcher-16 144B ± 0% 112B ± 0% -22.22% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
HeaderREMatcher-16 7.00 ± 0% 5.00 ± 0% -28.57% (p=0.000 n=10+10)
```
* httpcaddyfile: Fix unexpectedly removed policy
When user set on_demand tls option in a catch-all (:443) policy,
we expect other policies to not have the on_demand enabled
See ex in tls_automation_policies_5.txt
Btw, we can remove policies if they are **all** empty.
* Update caddyconfig/httpcaddyfile/tlsapp.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddyhttp: reverseproxy: fix hash selection policy
Fixes: #4135
Test: go test './...' -count=1
* caddyhttp: reverseproxy: add test to catch #4135
If you revert the last commit, the test will fail.
* caddyfile: Fix `import` replacing unrelated placeholders
See https://caddy.community/t/snippet-issue-works-outside-snippet/12231
So it turns out that `NewReplacer()` gives a replacer with some global defaults (like `{env.*}` and some system and time placeholders), which is not ideal when running `import` because we just want to replace `{args.*}` only, and nothing else.
* caddyfile: Add test
An idea that came up in https://caddy.community/t/save-internally-issued-wildcard-certificate-in-consul/11740, this a simple module that might be useful for anyone who uses storage modules that aren't filesystem, to let them load certs/keys externally issued for use by Caddy.
Bit goofy, since we need to fetch the certmagic.Storage during provisioning, it needs a wrapping struct instead of just being an array like `load_files`.
Future work might involve adding Caddyfile support via a subdirective of the `tls` directive maybe?
* caddyfile: reject recursive self-imports
* caddyfile: detect and reject cyclic imports of snippets and files
* caddyfile: do not be stickler about connected nodes not being connected already
* caddyfile: include missing test artifacts of cyclic imports
* address review comments
After reading a question about the `handle_response` feature of `reverse_proxy`, I realized that we didn't have a way of serving an arbitrary file with a status code other than 200. This is an issue in situations where you want to serve a custom error page in routes that are not errors, like the aforementioned `handle_response`, where you may want to retain the status code returned by the proxy but write a response with content from a file.
This feature is super simple, basically if a status code is configured (can be a status code number, or a placeholder string) then that status will be written out before serving the file - if we write the status code first, then the stdlib won't write its own (only the first HTTP status header wins).
Initial sd_notify support was added in #3963, but that sent signals from
both cmdRun and cmdReload. This approach has two drawbacks:
- Reloads initiated via the API do not send signals.
- The signals are sent from different processes, which requires the
`NotifyAccess=exec` directive in the unit file.
This change moves the NotifyReloading and NotifyReadiness invocations to
Load, which address both of those drawbacks. It also adds a
complimentary NotifyStopping method which is invoked from handleStop.
All the notify methods are defined in a notify package to avoid an
import loop.
As of go1.16, the `go` commands will no longer make automatic changes to go.{mod,sum} files (see: https://blog.golang.org/go116-module-changes). This broke the release script which relied on `go mod download` and/or `go build` to automatically generate the go.sum file. This commit explicitly invokes `go mod tidy` to have the go.sum file generated.
My editor automatically changed ioutil.ReadFile() to os.ReadFile() in accordance
with Go 1.16 changes. I didn't notice this until pushing.
But we still have to support Go 1.15 for a little while.
Caddy can now generate and persist its own instance ID, a UUID that is stored in
the data directory.
This makes it possible to differentiate it from other instances in a cluster.
* encode: implement prefer setting
* encode: minimum_length configurable via caddyfile
* encode: configurable content-types which to encode
* file_server: support precompressed files
* encode: use ReponseMatcher for conditional encoding of content
* linting error & documentation of encode.PrecompressedOrder
* encode: allow just one response matcher
also change the namespace of the encoders back, I accidently changed to precompressed >.>
default matchers include a * to match to any charset, that may be appended
* rounding of the PR
* added integration tests for new caddyfile directives
* improved various doc strings (punctuation and typos)
* added json tag for file_server precompress order and encode matcher
* file_server: add vary header, remove accept-ranges when serving precompressed files
* encode: move Suffix implementation to precompressed modules
* reverseproxy: Implement health_uri, replaces health_path, supports query
Also fixes a bug with `health_status` Caddyfile parsing , it would always only take the first character of the status code even if it didn't end with "xx".
* reverseproxy: Rename to URI, named logger, warn in Provision (for JSON)
This filter is intended to be useful in scenarios where you may want to
redact a value with a static string, giving you information that the
field did previously exist and was present, but not revealing the value
itself in the logs.
This was inspired by work on adding more complete support for removing
sensitive values from logs [1]. An example use case would be the
Authorization header in request log output, for which the value should
usually not be logged, but it may be quite useful for debugging to
confirm that the header was present in the request.
[1] https://github.com/caddyserver/caddy/issues/3958
This change is aimed at enhancing the logging module within the
Caddyfile directive to allow users to configure logs other than the HTTP
access log stream, which is the current capability of the Caddyfile [1].
The intent here is to leverage the same syntax as the server log
directive at a global level, so that similar customizations can be added
without needing to resort to a JSON-based configuration.
Discussion for this approach happened in the referenced issue.
Closes https://github.com/caddyserver/caddy/issues/3958
[1] https://caddyserver.com/docs/caddyfile/directives/log
* reverseproxy: Add duration/latency placeholders (close#4012) (and #2268)
Adds 4 placeholders, one is actually outside reverse proxy though:
{http.request.duration} is how long since the server decoded the HTTP request (headers).
{http.reverse_proxy.upstream.latency} is how long it took a proxy upstream to write the response header.
{http.reverse_proxy.upstream.duration} is total time proxying to the upstream, including writing response body to client.
{http.reverse_proxy.duration} is total time spent proxying, including selecting an upstream and retries.
Obviously, most of these are only useful at the end of a request, like when writing response headers or logs.
See also: https://caddy.community/t/any-equivalent-of-request-time-and-upstream-header-time-from-nginx/11418
* Add new placeholders to documentation
Proxy response bodies can now be buffered, and the size of the request body and
response body buffer can be limited. Any remaining content that doesn't fit in the
buffer will remain on the wire until it can be read; i.e. bodies are not truncated,
even if the buffer is not big enough.
This fulfills a customer requirement. This was made possible by their sponsorship!
Allows conveniently setting the resolvers for the DNS challenge using a TLS subdirective, which applies to default issuers, rather than having to explicitly define the issuers and overwrite the defaults.
The HTTP Caddyfile adapter can now configure the PKI app, and the acme_server directive can now be used to specify a custom CA used for issuing certificates. More customization options can follow later as needed.
If `tls <email>` is used, we should apply that to all applicable default issuers, not drop them. This refactoring applies implicit ACME issuer settings from the tls directive to all default ACME issuers, like ZeroSSL.
We also consolidate some annoying logic and improve config validity checks.
Ref: https://caddy.community/t/error-obtaining-certificate-after-caddy-restart/11335/8
* caddyhttp: Implement handler abort; new 'abort' directive (close#3871)
* Move abort directive ordering; clean up redirects
Seems logical for the end-all of handlers to go at the... end.
The Connection header no longer needs to be set there, since Close is
true, and the static_response handler now does that.
This commits dds 3 separate, but very related features:
1. Automated server identity management
How do you know you're connecting to the server you think you are? How do you know the server connecting to you is the server instance you think it is? Mutually-authenticated TLS (mTLS) answers both of these questions. Using TLS to authenticate requires a public/private key pair (and the peer must trust the certificate you present to it).
Fortunately, Caddy is really good at managing certificates by now. We tap into that power to make it possible for Caddy to obtain and renew its own identity credentials, or in other words, a certificate that can be used for both server verification when clients connect to it, and client verification when it connects to other servers. Its associated private key is essentially its identity, and TLS takes care of possession proofs.
This configuration is simply a list of identifiers and an optional list of custom certificate issuers. Identifiers are things like IP addresses or DNS names that can be used to access the Caddy instance. The default issuers are ZeroSSL and Let's Encrypt, but these are public CAs, so they won't issue certs for private identifiers. Caddy will simply manage credentials for these, which other parts of Caddy can use, for example: remote administration or dynamic config loading (described below).
2. Remote administration over secure connection
This feature adds generic remote admin functionality that is safe to expose on a public interface.
- The "remote" (or "secure") endpoint is optional. It does not affect the standard/local/plaintext endpoint.
- It's the same as the [API endpoint on localhost:2019](https://caddyserver.com/docs/api), but over TLS.
- TLS cannot be disabled on this endpoint.
- TLS mutual auth is required, and cannot be disabled.
- The server's certificate _must_ be obtained and renewed via automated means, such as ACME. It cannot be manually loaded.
- The TLS server takes care of verifying the client.
- The admin handler takes care of application-layer permissions (methods and paths that each client is allowed to use).\
- Sensible defaults are still WIP.
- Config fields subject to change/renaming.
3. Dyanmic config loading at startup
Since this feature was planned in tandem with remote admin, and depends on its changes, I am combining them into one PR.
Dynamic config loading is where you tell Caddy how to load its config, and then it loads and runs that. First, it will load the config you give it (and persist that so it can be optionally resumed later). Then, it will try pulling its _actual_ config using the module you've specified (dynamically loaded configs are _not_ persisted to storage, since resuming them doesn't make sense).
This PR comes with a standard config loader module called `caddy.config_loaders.http`.
Caddyfile config for all of this can probably be added later.
COMMITS:
* admin: Secure socket for remote management
Functional, but still WIP.
Optional secure socket for the admin endpoint is designed
for remote management, i.e. to be exposed on a public
port. It enforces TLS mutual authentication which cannot
be disabled. The default port for this is :2021. The server
certificate cannot be specified manually, it MUST be
obtained from a certificate issuer (i.e. ACME).
More polish and sensible defaults are still in development.
Also cleaned up and consolidated the code related to
quitting the process.
* Happy lint
* Implement dynamic config loading; HTTP config loader module
This allows Caddy to load a dynamic config when it starts.
Dynamically-loaded configs are intentionally not persisted to storage.
Includes an implementation of the standard config loader, HTTPLoader.
Can be used to download configs over HTTP(S).
* Refactor and cleanup; prevent recursive config pulls
Identity management is now separated from remote administration.
There is no need to enable remote administration if all you want is identity
management, but you will need to configure identity management
if you want remote administration.
* Fix lint warnings
* Rename identities->identifiers for consistency
Replaces the current Caddy executable with a new one from the build server. Honors custom builds, as long as plugins are registered on the Caddy website. Requires permissions to replace current executable, of course.
This is an experimental command that may get changed or removed later.
Previous commit improved the Caddyfile adapter so it doesn't unnecessarily add names to "skip" in "auto_https" when the server is already HTTP-only.
This commit updates the tests to reflect that change, while also fixing the Caddyfile formatting in many of the tests.
We also print the line number of the divergence between input and formatted version in Caddyfile adapt warnings - very useful for finding initial formatting problems.
This is probably an invasive change, but existing tests continue to pass.
It seems to make sense this way. There is likely an edge case I haven't
considered.
Allows user to disable OCSP stapling (including support in the Caddyfile via the ocsp_stapling global option) or overriding responder URLs. Useful in environments where responders are not reachable due to firewalls.
This changes the signature of UnmarshalGlobalFunc but this is probably OK since it's only used by this repo as far as we know.
We need this change in order to "remember" the previous value in case a global option appears more than once, which is now a possibility with the cert_issuer option since Caddy now supports multiple issuers in the order defined by the user.
Bonus: the issuer subdirective of tls now supports one-liner for "acme" when all you need to set is the directory:
issuer acme <dir>
* reverse_proxy: 1.health check headers can be set through Caddyfile using health_headers directive; 2.health check header host can be set properly
* reverse_proxy:
replace example with syntax definition
inline health_headers directive parse function
* bugfix: change caddyfile_adapt testcase file from space to tab
* reverseproxy: modify health_header value document as optional and add more test cases
* caddyfile: Introduce basic linting and fmt check
This will help encourage people to keep their Caddyfiles tidy.
* Remove unrelated tests
I am not sure that testing the output of warnings here is quite the
right idea; these tests are just for syntax and parsing success.
At some point we changed how paths are represented down the function calls of browse listings and forgot to update the canGoUp logic. I think this is right? It's simpler now.
This allows for finer-grained control when choosing alternate chains than
simply the previous/Certbot-esque behavior of "choose first chain that
contains an issuer's common name." This update allows you to sort by
length (if optimizing for efficiency on the wire) and also to select the
chain with a specific root CommonName.
The remote_ip matcher was reading the X-Forwarded-For header by default, but this behavior was not documented in anything that was released. This is also a less secure default, as it is trivially easy to spoof request headers. Reading IPs from that header should be optional, and it should not be the default.
This is technically a breaking change, but anyone relying on the undocumented behavior was just doing so by coincidence/luck up to this point since it was never in any released documentation. We'll still add a mention in the release notes about this.
Refactor redirect route creation into own function.
Improve condition for appending port.
Fixes a bug manifested through new test case:
TestAutoHTTPRedirectsWithHTTPListenerFirstInAddresses
* add integration test for null header matcher
* implement null header matcher syntax
* avoid repeating magic !
* check for field following ! character
* fastcgi: Set PATH_INFO to file matcher remainder as fallback
* fastcgi: Avoid changing scriptName when not necessary
* Stylistic tweaks
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
The docs at os/signal.Notify warn about this signal delivery loss bug at
https://golang.org/pkg/os/signal/#Notify, which says:
Package signal will not block sending to c: the caller must ensure
that c has sufficient buffer space to keep up with the expected signal
rate. For a channel used for notification of just one signal value,
a buffer of size 1 is sufficient.
Caught by a static analysis tool from Orijtech, Inc. called "sigchanyzer"
* fix(caddy): Avoid "operation was canceled" errors
- Also add error handling for StatusGatewayTimeout
* revert(caddy): Revert 504 handling
- This will potentially break load balancing and health checks
* Handle client cancellation as different error
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* acme_server: Refactor database creation apart from authority creation
This is a WIP commit that doesn't really offer anything other than
setting us up for using a UsagePool to gracefully reload acme_server
configs.
* Implement UsagePool
* Remove unused context
* Fix initializing non-ACME CA
This will handle cases where a DB is not provided
* Sanitize acme db path and clean debug logs
* Move regex to package level to prevent recompiling
* acme_server: switch to bbolt storage
There have been some issues with the badger storage engine
being used by the embedded acme_server. This will replace
the storage engine with bbolt
* Switch database path back to acme_server/db and remove if directory
* httpcaddyfile: First pass at implementing server options
* httpcaddyfile: Add listener wrapper support
* httpcaddyfile: Sort sbaddrs to make adapt output more deterministic
* httpcaddyfile: Add server options adapt tests
* httpcaddyfile: Windows line endings lol
* caddytest: More windows line endings lol (sorry Matt)
* Update caddyconfig/httpcaddyfile/serveroptions.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* httpcaddyfile: Reword listener address "matcher"
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* httpcaddyfile: Deprecate experimental_http3 option (moved to servers)
* httpcaddyfile: Remove validation step, no longer needed
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* ci: Use golangci's github action for linting
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix most of the staticcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the prealloc lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the misspell lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the varcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the errcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the bodyclose lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the deadcode lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the unused lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the gosec lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the gosimple lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the ineffassign lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the staticcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Revert the misspell change, use a neutral English
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Remove broken golangci-lint CI job
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Re-add errantly-removed weakrand initialization
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* don't break the loop and return
* Removing extra handling for null rootKey
* unignore RegisterModule/RegisterAdapter
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* single-line log message
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Fix lint after a1808b0dbf209c615e438a496d257ce5e3acdce2 was merged
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Revert ticker change, ignore it instead
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Ignore some of the write errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Remove blank line
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Use lifetime
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* close immediately
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Preallocate configVals
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddytls/distributedstek/distributedstek.go
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* reverseproxy: Enable error logging for connection upgrades
* reverseproxy: Change some of the error levels, unsugar
* Use unsugared log in one spot
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* implement default values for header directive
closes#3804
* remove `set_default` header op and rely on "require" handler instead
This has the following advantages over the previous attempt:
- It does not introduce a new operation for headers, but rather nicely
extends over an existing feature in the header handler.
- It removes the need to specify the header as "deferred" because it is
already implicitely deferred by the use of the require handler. This
should be less confusing to the user.
* add integration test for header directive in caddyfile
* bubble up errors when parsing caddyfile header directive
* don't export unnecessarily and don't canonicalize headers unnecessarily
* fix response headers not passed in blocks
* caddyfile: fix clash when using default header in block
Each header is now set in a separate handler so that it doesn't clash
with other headers set/added/deleted in the same block.
* caddyhttp: New idle_timeout default of 5m
* reverseproxy: fix random hangs on http/2 requests with server push (#3875)
see https://github.com/golang/go/issues/42534
* Refactor and cleanup with improvements
* More specific link
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
Co-authored-by: Денис Телюх <telyukh.denis@gmail.com>
Before, if there was an error in the error handler, we would not write a
status code, which resulted in Go writing a 200 for us by default, which
does not make sense when there's an error. Now we write the second
error's status if available, otherwise 500.
* caddytls: Support multiple issuers
Defaults are Let's Encrypt and ZeroSSL.
There are probably bugs.
* Commit updated integration tests, d'oh
* Update go.mod
* nitpicks and small improvements in basicauth module
1:
roll two if statements into one, since err will be nil in the second case anyhow
2:
unlock cache mutex after reading the key, as this happens by-value and reduces code complexity
3:
switch cache sync.Mutex to sync.RWMutex for better concurrency on cache fast track
* allocate the right kind of mutex
* fileserver: Improve and clarify file hiding logic
* Oops, forgot to run integration tests
* Make this one integration test OS-agnostic
* See if this appeases the Windows gods
* D'oh
Always follow the code path of hashing and comparing a plaintext
password even if the account is not found by the given username; this
ensures that similar CPU cycles are spent for both valid and invalid
usernames.
Thanks to @tylerlm for helping and looking into this!
* httpcaddyfile: Revise automation policy generation
This should fix a frustrating edge case where wildcard subjects are
used, which potentially get shadowed by more specific versions of
themselves; see the new tests for an example. This change is motivated
by an actual customer requirement.
Although all the tests pass, this logic is incredibly complex and
nuanced, and I'm worried it is not correct. But it took me about 4 days
to get this far on a solution. I did my best.
* Fix typo
We have users that have site blocks like *.*.tld with on-demand TLS
enabled. While *.*.tld does not qualify for a publicly-trusted cert due
to its wildcards, On-Demand TLS does not actually obtain a cert with
those wildcards, since it uses the actual hostname on the handshake.
This improves on that logic, but I am still not 100% satisfied with the
result since I think we need to also check if another site block is more
specific, like foo.example.tld, which might not have on-demand TLS
enabled, and make sure an automation policy gets created before the
more general policy with on-demand...
* reverseproxy: Fix dial placeholders, SRV, active health checks
Supercedes #3776
Partially reverts or updates #3756, #3693, and #3695
* reverseproxy: add integration tests
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* reverseproxy: fix breakage in handling SRV lookup introduced by 3695
* reverseproxy: validate against incompatible config options with lookup_srv
* reverseproxy: add integration test cases for validations involving lookup_srv
* reverseproxy: clarify the reason for skipping an iteration
* grammar.. Oxford comma
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Fixes#3753
* caddyfile: support vars and vars_regexp matchers in the caddyfile
* caddyfile: matchers: Brian Kernighan said printf is good debugging tool but didn't say keep them around
* metrics: Always track method label in uppercase
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Just use strings.ToUpper for clarity
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* metrics: Fixing panic while observing with bad exemplars
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Minor cleanup
The server is already added to the context. So, we can simply use that
to get the server name, which is a field on the server.
* Add integration test for auto HTTP->HTTPS redirects
A test like this would have caught the problem in the first place
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* reverseproxy: construct active health-check transport from scratch (Fixes#3691)
* reverseproxy: do upstream health-check on the correct alternative port
* reverseproxy: add integration test for health-check on alternative port
* reverseproxy: put back the custom transport for health-check http client
* reverseproxy: cleanup health-check integration test
* reverseproxy: fix health-check of unix socket upstreams
* reverseproxy: skip unix socket tests on Windows
* tabs > spaces
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* make the linter (and @francislavoie) happy
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* One more lint fix
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* httpcaddyfile: Ensure handle_path is sorted as equal to handle
* httpcaddyfile: Make mutual exclusivity grouping deterministic (I hope)
* httpcaddyfile: Add comment linking to the issue being fixed
* httpcaddyfile: Typo fix, comment clarity
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Update caddyconfig/httpcaddyfile/httptype.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* fileserver: Fix try_files for directories, windows fix
* fileserver: Add new file type placeholder, refactoring, tests
* fileserver: Review cleanup
* fileserver: Flip the return args order
* Fix-3585: added placeholder for a PEM encoded value of the certificate
* Update modules/caddyhttp/replacer.go
Change type of block and empty headers removed
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* fixed tests
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* logging: Implement Caddyfile support for filter encoder
* logging: Add support for parsing IP masks from strings
wip
* logging: Implement Caddyfile support for ip_mask
* logging: Get rid of unnecessary logic to allow strings, not that useful
* logging: Add adapt test
* Allow 'caddy fmt' to read from stdin
* fmt: use '-' as the file name for reading from stdin
* Minor adjustments
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
We recently introduced `if !cp.SettingsEmpty()` which conditionally
adds the connection policy to the list. If the condition evaluates to
false, the policy wouldn't actually be added, even if
hasCatchAllTLSConnPolicy was set to true on the previous line.
Now we set that variable in accordance with whether we actually add
the policy.
While debugging this I noticed that catch-all policies added early in
that loop (i.e. not at the end if we later determine we need one) are
not always at the end of the list. They should be, though, since they
are selected by which one matches first, and having a catch-all first
would nullify any more specific ones later in the list. So I added a
sort in consolidateConnPolicies to take care of that.
Should fix#3670 and
https://caddy.community/t/combining-on-demand-tls-with-custom-ssl-certs-doesnt-seem-to-work-in-2-1-1/9719
but I won't know for sure until somebody verifies it, since at least in
the GitHub issue there is not yet enough information (the configs are
redacted).
Now, a filename to hide that is specified without a path separator will
count as hidden if it appears in any component of the file path (not
only the last component); semantically, this means hiding a file by only
its name (without any part of a path) will hide both files and folders,
e.g. hiding ".git" will hide "/.git" and also "/.git/foo".
We also do prefix matching so that hiding "/.git" will hide "/.git"
and "/.git/foo" but not "/.gitignore".
The remaining logic is a globular match like before.
Update internal issuer for compatibility -- yay simpler code!
The .1 version also fixes non-critical SAN extensions that caused trust
issues on several clients.
* ci: Try Go 1.15 RC1 out of curiosity
* Go 1.15 was released; let's try it
* Update to latest quic-go
* Attempt at fixing broken test
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* httpcaddyfile: Flip `root` directive sort order
* httpcaddyfile: Sort directives with any matcher before those with none
* httpcaddyfile: Generalize reverse sort directives, improve logic
* httpcaddyfile: Fix "spelling" issue
* httpcaddyfile: Turns out the second change precludes the first
httpcaddyfile: Delete test that no longer makes sense
* httpcaddyfile: Shorten logic
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* caddytls: Add support for ZeroSSL; add Caddyfile support for issuers
Configuring issuers explicitly in a Caddyfile is not easily compatible
with existing ACME-specific parameters such as email or acme_ca which
infer the kind of issuer it creates (this is complicated now because
the ZeroSSL issuer wraps the ACME issuer)... oh well, we can revisit
that later if we need to.
New Caddyfile global option:
{
cert_issuer <name> ...
}
Or, alternatively, as a tls subdirective:
tls {
issuer <name> ...
}
For example, to use ZeroSSL with an API key:
{
cert_issuser zerossl API_KEY
}
For now, that still uses ZeroSSL's ACME endpoint; it fetches EAB
credentials for you. You can also provide the EAB credentials directly
just like any other ACME endpoint:
{
cert_issuer acme {
eab KEY_ID MAC_KEY
}
}
All these examples use the new global option (or tls subdirective). You
can still use traditional/existing options with ZeroSSL, since it's
just another ACME endpoint:
{
acme_ca https://acme.zerossl.com/v2/DV90
acme_eab KEY_ID MAC_KEY
}
That's all there is to it. You just can't mix-and-match acme_* options
with cert_issuer, because it becomes confusing/ambiguous/complicated to
merge the settings.
* Fix broken test
This test was asserting buggy behavior, oops - glad this branch both
discovers and fixes the bug at the same time!
* Fix broken test (post-merge)
* Update modules/caddytls/acmeissuer.go
Fix godoc comment
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Add support for ZeroSSL's EAB-by-email endpoint
Also transform the ACMEIssuer into ZeroSSLIssuer implicitly if set to
the ZeroSSL endpoint without EAB (the ZeroSSLIssuer is needed to
generate EAB if not already provided); this is now possible with either
an API key or an email address.
* go.mod: Use latest certmagic, acmez, and x/net
* Wrap underlying logic rather than repeating it
Oops, duh
* Form-encode email info into request body for EAB endpoint
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Bring `ensure_origin` and `origins` to caddyfile admin config
* Add unit test for caddyfile admin config update
* Add caddyfile adapt test for typical admin setup
* httpcaddyfile: Replace admin config error message when there's more arguments than needed
Replace d.Err() to d.ArgErr() since the latter provides similarly informative error message
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* ci: include tracking of GOOS for which Caddy fails to build
* ci: split cross-build check into separate workflow
* ci: cross-build check: make it clear the cross-build check is not a blocker
* ci: cross-build check: set annotation instead of failing the build
* ci: cross-build check: explicitly set continue-on-error to force success marker
* ci: cross-build check: send stderr to /dev/null
* ci: Simplify workflow names
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* fileserver: First attempt to fix failing test on Linux
I think I updated the wrong test case before
* Make new test function
I guess what we really are trying to test is the case insensitivity of
firstSplit. So a new test function is better for that.
We can't use a positional index on an original string that we got from
its lower-cased equivalent. Implement our own IndexFold() function b/c
the std lib does not have one.
* push: Implement HTTP/2 server push (close#3551)
* push: Abstract header ops by embedding into new struct type
This will allow us to add more fields to customize headers in
push-specific ways in the future.
* push: Ensure Link resources are pushed before response is written
* Change header name from X-Caddy-Push to Caddy-Push
* reverse proxy: Support more h2 stream scenarios (#3556)
* reverse proxy: add integration test for better h2 stream (#3556)
* reverse proxy: adjust comments as francislavoie suggests
* link to issue #3556 in the comments
* reverse proxy: add support for custom resolver
* reverse proxy: don't pollute the global resolver with bootstrap resolver setup
* Improve documentation of reverseproxy.UpstreamResolver fields
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* reverse proxy: clarify the name resolution conventions of upstream resolvers and bootstrap resolver
* remove support for bootstraper of resolver
* godoc and code-style changes
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
We already restore them within the retry loop, but after successful
proxy we didn't reset them, so as handlers bubble back up, they would
see the values used for proxying.
Thanks to @ziddey for identifying the cause.
* reverseproxy: Fix Caddyfile parsing for empty non-http transports
* Update modules/caddyhttp/reverseproxy/caddyfile.go
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Rename empty transport test
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* cel: fix validation of expression result type
The earlier code used the proto.Equals from github.com/gogo/protobuf, which failed to compare two messages of the same type for some reason. Switching to proto.Equal from the canonical github.com/golang/protobuf fixes the issue.
* deps: remove deprecated github.com/golang/protobuf in favor of google.golang.org/protobuf
* downgrade github.com/smallstep/nosql to resolve warning pb.proto warning
First try an exact lookup like before, but if it fails, strip the port
and try again. example.com:1234 should still use a logger keyed for
example.com if there is no key example.com:1234.
Catch-alls should always go last. Normally this is the case, but we have
a special case for comparing one wildcard-host site block to another
non-wildcard host site block; and a catch-all site block is also a
non-wildcard host site block, so now we have to special-case the
catch-all site block. Sigh.
This could be reproduced with a Caddyfile that has two site blocks:
":80" and "*.example.com", in that order.
* Adds global options for external account bindings
* Maybe other people use ctags too?
* Use nested block to configure external account
* go format files
* Restore acme_ca directive in test file
* Change Caddyfile config syntax for acme_eab
* Update test
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* ci: lay out foundation for s390x tests
* ci: uncomment the s390x test script & replace placeholders with real values
* ci: amend the s390x test job name to be more consistent with others
In commit f2ce81c, support for multiple path splitters was added. The
type of SplitPath changed from string to []string, and splitPos was
changed to loop through all values in SplitPath.
Before that commit, if SplitPath was empty, strings.Index returned 0 and
PATH_INFO was set correctly in buildEnv.
Currently, however, splitPos returns -1 for empty values of SplitPath,
behaving as if a split position could not be found at all. PATH_INFO is
then never set in buildEnv and remains empty.
Restore the old behaviour by explicitly checking whether SplitPath is
empty and returning 0 in splitPos.
Closes#3490
This is just a convenience if using a static_response handler in an
error route, by setting the default status code to the same one as
the error status.
Cache capacity is currently hard-coded at 1000 with random eviction.
It is enabled by default from Caddyfile configurations because I assume
this is the most common preference.
* caddyconfig: WIP implementation of handle_path
* caddyconfig: Complete the implementation - h.NewRoute was key
* caddyconfig: Add handle_path integration test
* caddyhttp: Use the path matcher as-is, strip the trailing *, update test
Correct behavior is not well defined because this is a non-standard
header field. This could be a "hop-by-hop" field much like
X-Forwarded-For is, but even our X-Forwarded-For implementation
preserves prior entries. Or, it could be best to preserve the original
value from the first hop, representing the protocol as facing the
client.
Let's try it the other way for a bit and see how it goes.
See https://caddy.community/t/caddy2-w-wordpress-behind-nginx-reverse-proxy/8174/3?u=matt
* add test case for SplitFrontMatter showing issue with windows newline
* fix issue with windows newline when using SplitFrontMatter
* Update modules/caddyhttp/templates/frontmatter.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* make it mere explicit what is trimmed from firstLine
* Update modules/caddyhttp/templates/frontmatter.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* fastcgi: Add new php_fastcgi subdirectives to override the shortcut
* fastcgi: Support "index off" to disable redir and try_files
* fastcgi: Remove whitespace to satisfy linter
* fastcgi: Run gofmt
* fastcgi: Make a new dispenser instead of using rewind
* fastcgi: Some fmt
* fastcgi: Add a couple adapt tests
* fastcgi: Clean up for loops
* fastcgi: Move adapt tests to separate files
* run: Add the possibility to load an env file
* run: change envfile flag var
* run: do not ignore err values
* Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* httpcaddyfile: Add shorthands for parameterized placeholders
httpcaddyfile: Now with regexp instead
httpcaddyfile: Allow dashes, gofmt
httpcaddyfile: Compile regexp only once
httpcaddyfile: Cleanup struct
httpcaddyfile: Optimize the replacers, pull out of the loop
httpcaddyfile: Add `{port}` shorthand
* httpcaddyfile: Switch `r.` to `re.`
* caddy: Add support for `d` duration unit
* Improvements to ParseDuration; add unit tests
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
Too flaky. We'll explore different avenues to testing s390x and ppc64le.
See discussion here: https://github.com/caddyserver/caddy/pull/3355
/cc @grooverdan, @Mohammed90 said he'll reach out to Elizabeth as you suggested.
* httpcaddyfile: Make global options pluggable
* httpcaddyfile: Add a global options adapt test
* httpcaddyfile: Wrap err
Co-Authored-By: Dave Henderson <dhenderson@gmail.com>
* httpcaddyfile: Revert wrap err
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
* adding wildcard matching of logger names
* reordering precedence for more specific loggers to match first
* removing dependence on certmagic and extra loop
Co-authored-by: GregoryDosh <GregoryDosh@users.noreply.github.com>
Closes#3365
* http: Add support in hash-password for reading from terminals/stdin
* FIXUP: Run gofmt -s
* FIXUP
* FIXUP: Apply suggestions from code review
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* FIXUP
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
While building a layer4 app for Caddy, I discovered that we need the
ability to fill a request's context just like the HTTP server does,
hence this exported function PrepareRequest().
* For `roll_size` and `roll_keep_for` directives, round up instead of down.
For example, if a user wants to be able to look back on 36 hours of logs,
but you must round to a 24-hour multiple, then it's better to round up to
48 hours (which includes the desired 36 hours) instead of down to 24 hours.
* `roll_size` had an off-by-one error that caused the size to be as much as
1 MB larger than requested. For example, requests of `1MB` and `1.1MB`
both became 2 MB. Now `1MB` means 1 MB, and `1.1MB` is rounded up to 2 MB.
https://caddy.community/t/cant-get-simple-alias-to-work/7911/8?u=matt
This removes an optimization where we amortized path matcher decoding.
The decoded matchers were index by... position... which obviously
changes during sorting. Duh.
Anyway, sorting is sliiightly slower now but the Caddyfile is not
really CPU-sensitive, so this is fine.
An upstream like https://localhost:80 is still forbidden, but an addr of
localhost:80 can be used while explicitly enabling TLS as an override;
we just don't allow the implicit behavior to be ambiguous.
When client certificate is enabled Caddy check only last certificate from
request. When this cert is not in list of trusted leaf certificates,
connection is rejected. According to RFC TLS1.x the sender's certificate
must come first in the list. Each following certificate must directly
certify the one preceding it.
This patch fix this problem - first certificate is checked instead of last.
This can lead to nicer, smaller JSON output for Caddyfiles like this:
a {
tls internal
}
b {
tls foo@bar.com
}
i.e. where the tls directive only configures automation policies, and
is merely meant to enable TLS on a server block (if it wasn't implied).
This helps keeps implicit config implicit.
Needs a little more testing to ensure it doesn't break anything
important.
* pki: Initial commit of embedded ACME server (#3021)
* reverseproxy: Support auto-managed TLS client certificates (#3021)
* A little cleanup after today's review session
Previously, matching by trying files other than the actual path of the
URI was:
file {
try_files <files...>
}
Now, the same can be done in one line:
file <files...>
As before, an empty file matcher:
file
still matches if the request URI exists as a file in the site root.
* reverse_proxy: Initial attempt at H2C transport/client support (#3218)
I have not tested this yet
* Experimentally enabling H2C server support (closes#3227)
See also #3218
I have not tested this
* reverseproxy: Clean up H2C transport a bit
* caddyhttp: Update godoc for h2c server; clarify experimental status
* caddyhttp: Fix trailers when recording responses (fixes#3236)
* caddyhttp: Tweak h2c config settings and docs
Moving to https://github.com/caddyserver/circuitbreaker
Nobody was using it anyway -- it works well, but something got fumbled
in a refactoring *months* ago. Turns out that we forgot the interface
guards AND botched a method name (my bad) - Ok() should have been OK().
So it would always have thrown a runtime panic if it tried to be loaded.
The module itself works well, but obviously nobody used it because
nobody reported the error. Fixing this while we move it to the new repo.
Removing this removes the last Bazaar/Launchpad dependency (I think).
* httpcaddyfile: Exclude access logs written to files from default log
Even though any logs can just be ignored, most users don't seem to like
configuring an access log to go to a file only to have it doubly appear
in the default log.
Related to:
- #3294
- https://caddy.community/t/v2-logging-format/7642/4?u=matt
- https://caddy.community/t/caddyfile-questions/7651/3?u=matt
* caddyhttp: General improvements to access log controls (fixes#3310)
* caddyhttp: Move log config nil check higher
* Rename LoggerName -> DefaultLoggerName
* matcher: Add `split_path` option to file matcher; used in php_fastcgi
* matcher: Skip try_files split if not the final part of the filename
* matcher: Add MatchFile tests
* matcher: Clarify SplitPath godoc
Sigh, apparently Linux is incapable of distinguishing host interfaces
in socket addresses, even though it works fine on Mac. I suppose we just
have to assume that any listeners with the same port are the same
address, completely ignoring the host interface on Linux... oh well.
* ci: Enable GoReleaser .deb support
* ci: Test .deb build
* ci: Fix typo
* ci: Turn off snapshot (breaks due to go mod edit)
* ci: Force the tag to rc3 for now
* ci: Let's try to publish the .debs
* ci: Attempt to enable build cache, rebuild after fixed line endings
* ci: Fix yml dupe ID issue, add caddy-api.service
* ci: Split cache keys between files so they're separate
* ci: Fix bindir
* ci: Update the script files
* ci: Retrigger
* ci: Push to gemfury
* ci: Use loop, fix bad env var
* ci: Retrigger
* ci: Try to force blank password?
* ci: Check if the token is actually present
* ci: Cleanup, remove debugging stuff
* ci: Remove useless comment
Panic would happen if an automation policy was specified in a singular
server block that had no hostnames in its address. Definitely an edge
case.
Fixed a bug related to checking for server blocks with a host-less key
that tried to make an automation policy. Previously if you had only two
server blocks like ":443" and another one at ":80", the one at ":443"
could not create a TLS automation policy because it thought it would
interfere with TLS automation for the block at ":80", but obviously that
key doesn't enable TLS because it is on the HTTP port. So now we are a
little smarter and count only non-HTTP-empty-hostname keys.
Also fixed a bug so that a key like "https://:1234" is sure to have TLS
enabled by giving it a TLS connection policy. (Relaxed conditions
slightly; the previous conditions were too strict, requiring there to be
a TLS conn policy already or a default SNI to be non-empty.)
Also clarified a comment thanks to feedback from @Mohammed90
* ci: Let's see if caching GOCACHE helps...
* ci: Use GOCACHE env instead (fixes windows), remove build -a
* ci: Hack to pull the GOCACHE env up to CI vars
* ci: Change cache key (mainly to wipe cache now)
* docs: Pull contributing document from v1 branch
* Update .github/CONTRIBUTING.md
Co-Authored-By: Matt Holt <mholt@users.noreply.github.com>
* docs: [Responsible -> Coordinated] Disclosure
* docs: Link to the new security policy page
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
To clarify, listening on wildcard interfaces is NOT the default and
should only be done under certain circumstances and when you know
what you're doing. Emits a warning in the log.
Fixes https://github.com/caddyserver/caddy-docker/issues/71
These functions are called at init-time, and their inputs are hard-coded
so there are no environmental or user factors that could make it fail
or succeed; the error return values are often ignored, and when they're
not, they are usually a fatal error anyway. To ensure that a programmer
mistake is not missed, we now panic instead.
Last breaking change 🤞
- Create two default automation policies; if the TLS app is used in
isolation with the 'automate' certificate loader, it will now use
an internal issuer for internal-only names, and an ACME issuer for
all other names by default.
- If the HTTP Caddyfile adds an 'automate' loader, it now also adds an
automation policy for any names in that loader that do not qualify
for public certificates so that they will be issued internally. (It
might be nice if this wasn't necessary, but the alternative is to
either make auto-HTTPS logic way more complex by scanning the names in
the 'automate' loader, or to have an automation policy without an
issuer switch between default issuer based on the name being issued
a certificate - I think I like the latter option better, right now we
do something kind of like that but at a level above each individual
automation policies, we do that switch only when no automation
policies match, rather than when a policy without an issuer does
match.)
- Set the default LoggerName rather than a LoggerNames with an empty
host value, which is now taken literally rather than as a catch-all.
- hostsFromKeys, the function that gets a list of hosts from server
block keys, no longer returns an empty string in its resulting slice,
ever.
Using html/template.HTML like we were doing before caused nested include
to be HTML-escaped, which breaks sites. Now we do not escape any of the
output; template input is usually trusted, and if it's not, users should
employ escaping actions within their templates to keep it safe. The docs
already said this.
Based on download stats, demand for 32-bit binaries these days is
extremely low. Also unify some of the filename conventions; just a
few bikeshedding changes :)
We now store the parsed site/server block keys with the server block,
rather than parsing the addresses every time we read them.
Also detect conflicting schemes, i.e. TLS and non-TLS cannot be served
from the same server (natively -- modules could be built for it).
Also do not add site subroutes (subroutes generated specifically from
site blocks in the Caddyfile) that are empty.
Thus far the fuzzers have found a few crashers in the Caddyfile parser. However, the fuzzer have been stuck at import glob expansion after import glob expansion, which aren't reproducible.
Certificate selection used to be a module, but this seems unnecessary,
especially since the built-in CustomSelectionPolicy allows quite complex
selection logic on a number of fields in certs. If we need to extend
that logic, we can, but I don't think there are SO many possibilities
that we need modules.
This update also allows certificate selection to choose between multiple
matching certs based on client compatibility and makes a number of other
improvements in the default cert selection logic, both here and in the
latest CertMagic.
The hardest part of this was the conn policy consolidation logic
(Caddyfile only, of course). We have to merge connection policies that
we can easily combine, because if two certs are manually loaded in a
Caddyfile site block, that produces two connection policies, and each
cert is tagged with a different tag, meaning only the first would ever
be selected. So given the same matchers, we can merge the two, but this
required improving the Tag selection logic to support multiple tags to
choose from, hence "tags" changed to "any_tag" or "all_tags" (but we
use any_tag in our Caddyfile logic).
Combining conn policies with conflicting settings is impossible, so
that should return an error if two policies with the exact same matchers
have non-empty settings that are not the same (the one exception being
any_tag which we can merge because the logic for them is to OR them).
It was a bit complicated. It seems to work in numerous tests I've
conducted, but we'll see how it pans out in the release candidates.
If a placeholder in the path component injects a query string such as
the {http.request.uri} placeholder is wont to do, we need to separate it
out from the path.
If a site block has a key like "http://localhost:2016", then the log for
that site must be mapped to "localhost:2016" and not just "localhost"
because "localhost:2016" will be the value of the Host header of requests.
But a key like "localhost:80" does not include the port since the Host
header will not include ":80" because it is a standard port.
Fixes https://caddy.community/t/v2-common-log-format-not-working/7352?u=matt
See https://caddy.community/t/v2-match-any-path-but-files/7326/8?u=matt
If rewrites (or redirects, for that matter) match on file existence,
the file matcher would need to know the root of the site.
Making this change implies that root directives that depend on rewritten
URIs will not work as expected. However, I think this is very uncommon,
and am not sure I have ever seen that. Usually, dynamic roots are based
on host, not paths or query strings.
I suspect that rewrites based on file existence will be more common than
roots based on rewritten URIs, so I am moving root to be the first in
the list.
Users can always override this ordering with the 'order' global option.
Either Dial or LookupSRV will be set, but if we rely on Dial always
being set, we could run into bugs.
Note: Health checks don't support SRV upstreams.
* tls: Support placeholders in key_type
* caddytls: Simplify placeholder support for ap.KeyType
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* chore: ci: preliminary CD support
* chore: ci: split release process into its own workflow
* chore: ci: cleanup the ci.yml and .goreleaser.yml
* chore: ci: unshallowify the clone before searching for the closes tag
* chore: tidy up goreleaser config & the release githubaction
* chore: add --no-tty to gpg args
* chore: more gpg args
* chore: try with default gpg args by goreleaser
* chore: gpg...
* chore: set GPG_TTY
* chore: preset gpg conf
* Apply suggestions from code review
chore: tidy up the .goreleaser.yml
Co-Authored-By: Dave Henderson <dhenderson@gmail.com>
* chore: gpg debugging
* chore: set and export the tty for gpg
* chore: gpg..
* chore: use the exact same line from goreleaser-action README for singing
* chore: remove signing stanzas from ymls
* chore: clean up the release action for final submission
* quote the arguments of echo
Co-Authored-By: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* run caddy tests in process
* call main with run args
* exclude tests - windows
* include json example
* disable caddyfile tests, include json test with non trusted local ca
* converted SNI tests to json syntax
* admin: Refactor /load endpoint out of caddy package
This eliminates the caddy package's dependency on the caddyconfig
package, which helps prevent import cycles.
* v2: adapter: register config adapters as Caddy modules
* v2: adapter: simplify adapter registration as adapters and modules
* v2: adapter: let RegisterAdapter be in charge of registering adapters as modules
* v2: adapter: remove underscrores placeholders
* v2: adapter: explicitly ignore the error of writing response of writing warnings back to client
* Implicitly wrap config adapters as modules
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
The comments in the code should explain the new logic thoroughly.
The basic problem for the issue was that we were overriding a catch-all
automation policy's explicitly-configured issuer with our own, for names
that we thought looked like public names. In other words, one could
configure an internal issuer for all names, but then our auto HTTPS
would create a new policy for public-looking names that uses the
default ACME issuer, because we assume public<==>ACME and
nonpublic<==>Internal, but that is not always the case. The new logic
still assumes nonpublic<==>Internal (on catch-all policies only), but
no longer assumes that public-looking names always use an ACME issuer.
Also fix a bug where HTTPPort and HTTPSPort from the HTTP app weren't
being carried through to ACME issuers properly. It required a bit of
refactoring.
* WIP: Trying to make a new branch
* Create fuzzing.yml
* Update ci.yml
* Try using reviewdog for golangci-lint
* Only run lint on ubuntu
* Whoops, wrong matrix variable
* Let's try just ubuntu for the moment
* Remove integration tests
* Let's see what the tree looks like (where's the binary)
* Let's plant a tree
* Let's look at another tree
* Burn the tree
* Let's build in the right dir
* Turn on publishing artifacts
* Add gobin to path
* Try running golangci-lint earlier
* Try running golangci-lint on its own, with checkout@v1
* Try moving golangci-lint back into ci.yml as a separate job
* Turn off azure-pipelines
* Remove the redundant name, see how it looks
* Trim down the naming some more
* Turn on windows and mac
* Try to fix windows build, cleanup
* Try to fix strange failure on windows
* Print our the coerce reason
* Apparently $? is 'True' on Windows, not 1 or 0
* Try setting CGO_ENABLED as an env in yml
* Try enabling/fixing the fuzzer
* Print out github event to check, fix step name
* Fuzzer needs the code
* Add GOBIN to PATH for fuzzer
* Comment out fork condition, left in-case we want it again
* Remove obsolete comment
* Comment out the coverage/test conversions for now
* Set continue-on-error: true for fuzzer, it runs out of mem
* Add some clarification to the retained commented sections
Adds `Alt-Svc` to the list of headers that get removed when proxying
to a backend.
This fixes the issue of having the contents of the Alt-Svc header
duplicated when proxying to another Caddy server.
* caddyhttp: Implement CEL matcher (see #3051)
CEL (Common Expression Language) is a very fast, flexible way to express
complex logic, useful for matching requests when the conditions are not
easy to express with JSON.
This matcher may be considered experimental even after the 2.0 release.
* Improve CEL module docs
* rewrite: strip_prefix, strip_suffix, uri_replace -> uri (closes#3140)
* Add period, to satisfy @whitestrake :) and my own OCD
* Restore implied / prefix
It's hard to say whether this was actually a bug, but the linked issue
shows why the old behavior was confusing. Basically, we infer that a
rewrite handler is supposed to act as an internal redirect, which likely
means it will no longer match the matcher(s) it did before the rewrite.
So if the rewrite directive shares a matcher with any adjacent route or
directive, it can be confusing/misleading if we consolidate the rewrite
into the same route as the next handler, which shouldn't (probably) match
after the rewrite is complete.
This is kiiiind of a hacky workaround to a quirky problem.
For edge cases like these, it is probably "cleaner" to just use handle
blocks instead, to group handlers under the same matcher, nginx-style.
* added sni tests
* set the default sni when there is no host to match
* removed invalid sni test. Disabled tests that rely on host headers.
* readded SNI tests. Added logging of config load times
Wrapping listeners is useful for composing custom behavior related
to accepting, closing, reading/writing connections (etc) below the
application layer; for example, the PROXY protocol.
When using the default automation policy specifically, ap.Issuer would
be nil, so we'd end up overwriting the ap.magic.Issuer's default value
(after New()) with nil; this instead sets Issuer on the template before
New() is called, and no overwriting is done.
* add integration tests
* removed SNI test
* remove integration test condition
* minor edit
* fix sni when using static certificates
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* pki: Initial commit of PKI app (WIP) (see #2502 and #3021)
* pki: Ability to use root/intermediates, and sign with root
* pki: Fix benign misnamings left over from copy+paste
* pki: Only install root if not already trusted
* Make HTTPS port the default; all names use auto-HTTPS; bug fixes
* Fix build - what happened to our CI tests??
* Fix go.mod
It's still not perfect but I think it should be more correct for
slightly more complex configs. Might still fall apart for complex
configs that use on-demand TLS or at a large scale (workarounds are
to just implement your own redirects, very easy to do anyway).
Fixes#3116
* Rework Replacer loop to ignore escaped braces
* Add benchmark tests for replacer
* Optimise handling of escaped braces
* Handle escaped closing braces
* Remove additional check for closing brace
This commit removes the additional check for input in which the closing
brace appears before the opening brace. This check has been removed for
performance reasons as it is deemed an unlikely edge case.
* Check for escaped closing braces in placeholder name
* ability to specify that client cert must be present in SSL
* changed the clientauthtype to string and make room for the values supported by go as in caddy1
* renamed the config parameter according to review comments and added documentation on allowed values
* missed a reference
* Minor cleanup; docs enhancements
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
This is a breaking change primarily in two areas:
- Storage paths for certificates have changed
- Slight changes to JSON config parameters
Huge improvements in this commit, to be detailed more in
the release notes.
The upcoming PKI app will be powered by Smallstep libraries.
* remove the certificate tag tracking from global state
* refactored helper state, added log counter
* moved state initialisation close to where it is used.
* added helper state comment
Previously the formatter did not include support for
blocks inside other blocks. Hence the formatter could
not indent some files properly. This fixes it.
Fixes#3104
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
This takes the config file as input and formats it.
Prints the result to stdout. Can write changes to
file if `--write` flag is passed.
Fixes#3020
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
This makes it more convenient to configure quick proxies that use HTTPS
but also introduces a lot of logical complexity. We have to do a lot of
verification for consistency and errors.
Path and query string is not supported (i.e. no rewriting).
Scheme and port can be inferred from each other if HTTP(S)/80/443.
If omitted, defaults to HTTP.
Any explicit transport config must be consistent with the upstream
schemes, and the upstream schemes must all match too.
But, this change allows a config that used to require this:
reverse_proxy example.com:443 {
transport http {
tls
}
}
to be reduced to this:
reverse_proxy https://example.com
which is really nice syntactic sugar (and is reminiscent of Caddy 1).
Small expansion to the work done in https://github.com/caddyserver/caddy/pull/2963 which simply calls `os.ExpandEnv` so env vars like `{$URL}` where `$URL=$SCHEME://$HOST:$PORT` (contrived) get the expanded $SCHEME, $HOST, and $PORT variables included
* httpcaddyfile: Begin implementing log directive, and debug mode
For now, debug mode just sets the log level for all logs to DEBUG
(unless a level is specified explicitly).
* httpcaddyfile: Finish 'log' directive
Also rename StringEncoder -> SingleFieldEncoder
* Fix minor bug in replacer (when vals are empty)
* caddytls: Add CipherSuiteName and ProtocolName functions
The cipher_suites.go file is derived from a commit to the Go master
branch that's slated for Go 1.14. Once Go 1.14 is released, this file
can be removed.
* caddyhttp: Use commonLogEmptyValue in common_log replacer
* caddyhttp: Add TLS placeholders
* caddytls: update unsupportedProtocols
Don't export unsupportedProtocols and update its godoc to mention that
it's used for logging only.
* caddyhttp: simplify getRegTLSReplacement signature
getRegTLSReplacement should receive a string instead of a pointer.
* caddyhttp: Remove http.request.tls.client.cert replacer
The previous behavior of printing the raw certificate bytes was ported
from Caddy 1, but the usefulness of that approach is suspect. Remove
the client cert replacer from v2 until a use case is presented.
* caddyhttp: Use tls.CipherSuiteName from Go 1.14
Remove ported version of CipherSuiteName in the process.
* Add handler for unhandled errors in errorChain
Currently, when an error chain is defined, the default error handler is
bypassed entirely - even if the error chain doesn't handle every error.
This results in pages returning a blank 200 OK page.
For instance, it's possible for an error chain to match on the error
status code and only handle a certain subtype of errors (like 403s). In
this case, we'd want any other errors to still go through the default
handler and return an empty page with the status code.
This PR changes the "suffix handler" passed to errorChain.Compile to
set the status code of the response to the error status code.
Fixes#3053
* Move the errorHandlerChain middleware to variable
* Style fix
* Fix crash when specifying "*" to header directive.
Fixes#3060
* Look Host header in header and header_regexp.
Also, if more than one header is provided, header_regexp now looks for
extra headers values to reflect the behavior from header.
Fixes#3059
* Fix parsing of named header_regexp in Caddyfile.
See #3059
See end of issue #3004. Loading the same certificate file multiple times
with different tags will result in it being de-duplicated in the in-
memory cache, because of course they all have the same bytes. This
meant that any certs of the same filename loaded with different tags
would be overwritten by the next certificate of the same filename, and
any conn policies looking for the tags of the previous ones would never
find them, causing connections to fail.
So, now we remember cert filenames and their tags, instead of loading
them multiple times and overwriting previous ones.
A user crafting their own JSON might make this error too... maybe we
won't see it happen. But if it does, one possibility is, when loading
a duplicate cert, instead of discarding it completely, merge the tag
list into the one that's already stored in the cache, then discard.
The documentation specifies that the hash algorithm defaults to bcrypt.
However, the implementation returns an error in provision if no hash is
provided.
Fix this inconsistency by *actually* defaulting to bcrypt.
Configuration via the Caddyfile requires use of env variables, but
an upstream issue is currently blocking that:
https://github.com/go-acme/lego/issues/1054
Providers will need to be retrofitted upstream in order to support env
var configuration.
We don't load the provider directly, because the lego provider types
aren't designed for JSON configuration and they are not implemented
as Caddy modules (there are some setup steps which a Provision call
would need to do, but they do not have Provision methods, they have
their own constructor functions that we have to wrap).
Instead of loading the challenge providers directly, the modules are
simple wrappers over the challenge providers, to facilitate the JSON
config structure and to provide a consistent experience. This also lets
us swap out the underlying challenge providers transparently if needed;
it acts as a layer of abstraction.
This is temporary as we prepare for a stable v2 release. We don't want
to make promises we don't know we can keep, and the Starlark integration
deserves much more focused attention which resources and funding do not
currently permit. When the project is financially stable, I will be able
to revisit this properly and add flexible, robust Starlark scripting
support to Caddy 2.
If user provides their own certs or makes any hostname-specific TLS
connection policy, it means that no TLS connection would be served for
any other hostnames, even though you'd expect that TLS is enabled for
them, too. So now we append a catch-all conn policy if none exist, which
allows all ClientHellos to be matched and served.
We also fix the consolidation of automation policies, which previously
gobbled up automation policies without hosts in favor of automation
policies with hosts. Instead of a host-specific policy eating up an
identical catch-all policy, the catch-all policy eats up the identical
host-specific policy, ensuring that the policy is applied to all hosts
which need it.
See also:
https://caddy.community/t/v2-automatic-https-certificate-errors/6847/9?u=matt
This ensure that if there are multiple certs that match a particular
ServerName or other parameter, then specifically the one the user
provided in the Caddyfile will be used.
This is necessary to avoid a race for sockets. Both the HTTP servers and
CertMagic solvers will try to bind the HTTP/HTTPS ports, but we need to
make sure that our HTTP servers bind first. This is kind of a new thing
now that management is async in Caddy 2.
Also update to CertMagic 0.9.2, which fixes some async use cases at
scale.
See https://caddy.community/t/caddy-server-that-returns-only-ip-address-as-text/6928/6?u=matt
In most cases, we will want to apply header operations immediately,
rather than waiting until the response is written. The exceptions are
generally going to be if we are deleting a header field or if a field is
to be overwritten. We now automatically defer header ops if deleting a
header field, and allow the user to manually enable deferred mode with
the defer subdirective.
Paths always begin with a slash, and omitting the leading slash could be
convenient to avoid confusion with a path matcher in the Caddyfile. I do
not think there would be any harm to implicitly add the leading slash.
Before, listener ports could be wrong because ParseAddress doesn't know
about the user-configured HTTP/HTTPS ports, instead hard-coding port 80
or 443, which could be wrong if the user changed them to something else.
Now we defer port and scheme validation/inference to a later part of
building the output JSON.
* v2: add documentation for circuit breaker config and "random selection" load balancing policy
* v2: rename circuit breaker config inline key from `type` to `breaker` to avoid json key clash between the `circuit_breaker` type and the `type` field of the generic circuit breaker Config struct used by circuit breaking implementations
* v2: restore the circuit breaker inline key to `type` and rename the name circuit breaker config field from `Type` to `Factor`
Using rewrite is like saying, "I accept this request, but I just need
to act on it as if it came in differently."
Whereas redir implies more of, "I reject this request, send it to me
differently, then I will process it."
Makes sense for it to come before rewrites. This can always be changed
using the 'order' global option if needed.
The fix that was initially put forth in #2971 was good, but only for
up to one layer of nesting. The real problem was that we forgot to
increment nesting when already inside a block if we saw another open
curly brace that opens another block (dispenser.go L157-158).
The new 'handle' directive allows HTTP Caddyfiles to be designed more
like nginx location blocks if the user prefers. Inside a handle block,
directives are still ordered just like they are outside of them, but
handler blocks at a given level of nesting are mutually exclusive.
This work benefitted from some refactoring and cleanup.
This allows individual directives to be ordered relative to others,
where order matters (for example HTTP handlers). Will primarily be
useful when developing new directives, so you don't have to modify the
Caddy source code. Can also be useful if you prefer that redir comes
before rewrite, for example. Note that these are global options. The
route directive can be used to give a specific order to a specific group
of HTTP handler directives.
In the v1 Caddyfile, only the first matching site definition would be
used, so setting these `Terminal: true` ensures that only the first
matching one is used in v2, too.
We also have to sort by key specificity... Caddy 1 had a special data
structure for selecting the most specific site definition, but we don't
have that structure in v2, so we need to sort by length (of host and
path, separately). For blocks where more than one key is present, we
choose the longest host and path (independently, need not be from same
key) by which to sort.
Before, modifying the path might have affected how a new query string
was built if the query string relied on the path. Now, we build each
component in isolation and only change the URI on the request later.
Also, prevent trailing & in query string.
This splits automatic HTTPS into two phases. The first provisions the
route matchers and uses them to build the domain set and configure
auto HTTP->HTTPS redirects. This happens before the rest of the
provisioning does.
The second phase takes place at the beginning of the app start. It
attaches pointers to the tls app to each server, and begins certificate
management for the domains that were found in the first phase.
Our new parser also preserves original parameter order, rather than
re-encoding using the std lib (which sorts).
The renamed parameters are a breaking change but they're new enough
that I don't think anyone is using them.
When we append a token to the new dispenser, we need to consume it in the parent, too; otherwise it gets scanned twice, which in this case messed up the nesting count which got decremented once too many times.
* fix replacing variables on imported files
* refactored replaceEnvVars to ensure it is always called
* Use byte slices for easier use
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
This is because of our sequential handling logic which was recently
merged; if vars is the first handler in the chain, it will be run before
the next route's matchers are executed, so there's no need to nest the
handlers anymore.
* http: path matcher: exact match by default; substring matches (#2959)
This is a breaking change.
* caddyfile: Change "matcher" directive to "@matcher" syntax (#2959)
* cmd: Assume caddyfile adapter for config files named Caddyfile
* Sub-sort handlers by path matcher length (#2959)
Caddyfile-generated subroutes have handlers, which are sorted first by
directive order (this is unchanged), but within directives we now sort
by specificity of path matcher in descending order (longest path first,
assuming that longest path is most specific).
This only applies if there is only one matcher set, and the path
matcher in that set has only one path in it. Path matchers with two or
more paths are not sorted like this; and routes with more than one
matcher set are not sorted like this either, since specificity is
difficult or impossible to infer correctly.
This is a special case, but definitely a very common one, as a lot of
routing decisions are based on paths.
* caddyfile: New 'route' directive for appearance-order handling (#2959)
* caddyfile: Make rewrite directives mutually exclusive (#2959)
This applies only to rewrites in the top-level subroute created by the
HTTP caddyfile.
Previously, all matchers in a route would be evaluated before any
handlers were executed, and a composite route of the matching routes
would be created. This made rewrites especially tricky, since the only
way to defer later matchers' evaluation was to wrap them in a subroute,
or to invoke a "rehandle" which often caused bugs.
Instead, this new sequential design evaluates each route's matchers then
its handlers in lock-step; matcher-handlers-matcher-handlers...
If the first matching route consists of a rewrite, then the second route
will be evaluated against the rewritten request, rather than the original
one, and so on.
This should do away with any need for rehandling.
I've also taken this opportunity to avoid adding new values to the
request context in the handler chain, as this creates a copy of the
Request struct, which may possibly lead to bugs like it has in the past
(see PR #1542, PR #1481, and maybe issue #2463). We now add all the
expected context values in the top-level handler at the server, then
any new values can be added to the variable table via the VarsCtxKey
context key, or just the GetVar/SetVar functions. In particular, we are
using this facility to convey dial information in the reverse proxy.
Had to be careful in one place as the middleware compilation logic has
changed, and moved a bit. We no longer compile a middleware chain per-
request; instead, we can compile it at provision-time, and defer only the
evaluation of matchers to request-time, which should slightly improve
performance. Doing this, however, we take advantage of multiple function
closures, and we also changed the use of HandlerFunc (function pointer)
to Handler (interface)... this led to a situation where, if we aren't
careful, allows one request routed a certain way to permanently change
the "next" handler for all/most other requests! We avoid this by making
a copy of the interface value (which is a lightweight pointer copy) and
using exclusively that within our wrapped handlers. This way, the
original stack frame is preserved in a "read-only" fashion. The comments
in the code describe this phenomenon.
This may very well be a breaking change for some configurations, however
I do not expect it to impact many people. I will make it clear in the
release notes that this change has occurred.
* transform a caddyfile with environment variables
* support adapt time and runtime variables in the caddyfile
* caddyfile: Pre-process environment variables before parsing
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Allows specifying ca certs with by filename in
`reverse_proxy.transport`.
Example
```
reverse_proxy /api api:443 {
transport http {
tls
tls_trusted_ca_certs certs/rootCA.pem
}
}
```
Config auto-saving is on by default and can be disabled. The --environ
flag (or environ subcommand) now print more useful information from
Caddy and the runtime, including some nifty paths.
It seems silly to have to add a single, empty TLS connection policy to
a server to enable TLS when it's only listening on the HTTPS port. We
now do this for the user as part of automatic HTTPS (thus, it can be
disabled / overridden).
See https://caddy.community/t/v2-catch-all-server-with-automatic-tls/6692/2?u=matt
This commit goes a long way toward making automated documentation of
Caddy config and Caddy modules possible. It's a broad, sweeping change,
but mostly internal. It allows us to automatically generate docs for all
Caddy modules (including future third-party ones) and make them viewable
on a web page; it also doubles as godoc comments.
As such, this commit makes significant progress in migrating the docs
from our temporary wiki page toward our new website which is still under
construction.
With this change, all host modules will use ctx.LoadModule() and pass in
both the struct pointer and the field name as a string. This allows the
reflect package to read the struct tag from that field so that it can
get the necessary information like the module namespace and the inline
key.
This has the nice side-effect of unifying the code and documentation. It
also simplifies module loading, and handles several variations on field
types for raw module fields (i.e. variations on json.RawMessage, such as
arrays and maps).
I also renamed ModuleInfo.Name -> ModuleInfo.ID, to make it clear that
the ID is the "full name" which includes both the module namespace and
the name. This clarity is helpful when describing module hierarchy.
As of this change, Caddy modules are no longer an experimental design.
I think the architecture is good enough to go forward.
Adds tests for both the path matcher and host matcher for case
insensitivity.
If case sensitivity is required for the path, a regexp matcher can
be used instead.
This is the v2 equivalent fix of PR #2882.
* fix OOM issue caught by fuzzing
* use ParsedAddress as the struct name for the result of ParseNetworkAddress
* simplify code using the ParsedAddress type
* minor cleanups
* Add support for placeholders in Config
Fixes#2870
* Replace placeholders only in logging config.
Placeholders in log level and filename incase of file output are replaced.
* Add Provision to filewriter module for replacing placeholders
Errors in the 4xx range are client errors, and they don't need to be
entered into the server's error logs. 4xx errors are still recorded in
the access logs at the error level.
This makes it easier to make "standard" caddy builds, since you'll only
need to add a single import to get all of Caddy's standard modules.
There is a package for all of Caddy's standard modules (modules/standard)
and a package for the HTTP app's standard modules only
(modules/caddyhttp/standard).
We still need to decide which of these, if not all of them, should be
kept in the standard build. Those which aren't should be moved out of
this repo. See #2780.
First evidenced in #2658, listener deadlines would sometimes be set
after clearing them, resulting in endless i/o timeout errors, which
leave all requests hanging. This bug is fixed by synchronizing the
calls to SetDeadline: when Close() is called, the deadline is first
set to a time in the past, and the lock is released only after the
deadline is set, so when the other servers break out of their Accept()
calls, they will clear the deadline *after* it was set. Before, the
clearing could sometimes come before the set, which meant that it was
left in a timeout state indefinitely.
This may not yet be a perfect solution -- ideally, the setting and
clearing of the deadline would happen exactly once per underlying
listener, not once per fakeCloseListener, but in rigorous testing with
these changes (comprising tens of thousands of config reloads), I was
able to verify that no race condition is manifest.
Fixed several bugs and made other improvements. All config changes are
now mediated by the global config state manager. It used to be that
initial configs given at startup weren't tracked, so you could start
caddy with --config caddy.json and then do a GET /config/ and it would
return null. That is fixed, along with several other general flow/API
enhancements, with more to come.
* Always cleanup admin endpoint first
* Error out if no config has been set (#2833)
* Ignore explicitly missing admin config (#2833)
* Separate config loading from admin initialization (#2833)
* Add admin option to specify admin listener address (#2833)
* Use zap for reporting admin endpoint status
* fuzz: add missing fuzzer by fixing .gitignore adding a negation for caddyfile/ directory
* ci: print fuzzing type for debuggability and traceability
* README: update the Fuzzit badge to point to the correct Caddy server Github organization
Not really necessary; underlying work is done by json.Unmarshal which
is part of the Go standard lib. Also, it called Run, which potentially
tries to get certificates; we should not let that happen.
* logging: Initial implementation
* logging: More encoder formats, better defaults
* logging: Fix repetition bug with FilterEncoder; add more presets
* logging: DiscardWriter; delete or no-op logs that discard their output
* logging: Add http.handlers.log module; enhance Replacer methods
The Replacer interface has new methods to customize how to handle empty
or unrecognized placeholders. Closes#2815.
* logging: Overhaul HTTP logging, fix bugs, improve filtering, etc.
* logging: General cleanup, begin transitioning to using new loggers
* Fixes after merge conflict
* fuzz-ci: fix the authentication call for fuzzit by using the --api-key flag rather than the `auth` command
* Allow fuzzing on schedules as well as non-fork PRs
Closes#2710
* fuzz: lay down the foundation for continuous fuzzing
* improve the fuzzers and add some
* fuzz: add Fuzzit badge to README & enable fuzzers submission in CI
* v2-fuzz: do away with the submodule approach for fuzzers
* fuzz: enable fuzzit
* v2: speed up some of powershell's processes
* v2-ci: downloading latest Go on Windows isn't slow anymore, so update the log message accordingly
* v2: CI: use 7z on Windows instead of Expand-Archive
* file_server: Make tests work on Windows
* caddyfile: Fix escaping when character is not escapable
We only escape certain characters depending on inside or outside of
quotes (mainly newlines and quotes). We don't want everyone to have to
escape Windows file paths like C:\\Windows\\... but we can't drop the
\ either if it's just C:\Windows\...
* v2: split golangci-lint configuration into its own file to allow code editors to take advantage of it
* v2: simplify code
* v2: set the correct lint output formatting
* v2: invert the logic of linter's configuration of output formatting to allow the editor convenience over CI-specific customization. Customize the output format in CI by passing the flag.
* v2: remove irrelevant golangci-lint config
This PR enables the use of placeholders in an upstream's Dial address.
A Dial address must represent precisely one socket after replacements.
See also #998 and #1639.
This implements HTTP basicauth into Caddy 2. The basic auth module will
not work with passwords that are not securely hashed, so a subcommand
hash-password was added to make it convenient to produce those hashes.
Also included is Caddyfile support.
Closes#2747.
This migrates a feature that was previously reserved for enterprise
users, according to #2786.
The Starlark integration needs to be updated since this was made before
some significant changes in the v2 code base. When functional, it makes
it possible to have very dynamic HTTP handlers. This will be a long-term
ongoing project.
Credit to Danny Navarro
This migrates a feature that was previously reserved for enterprise
users, according to https://github.com/caddyserver/caddy/issues/2786.
Custom certificate selection policies allow advanced control over which
cert is selected when multiple qualify to satisfy a TLS handshake.
This migrates a feature that was previously reserved for enterprise
users, according to https://github.com/caddyserver/caddy/issues/2786.
TLS session ticket keys are sensitive, so they should be rotated on a
regular basis. Only Caddy does this by default. However, a cluster of
servers that rotate keys without synchronization will lose the benefits
of having sessions in the first place if the client is routed to a
different backend. This module coordinates STEK rotation in a fleet so
the same keys are used, and rotated, across the whole cluster. No other
server does this, but Twitter wrote about how they hacked together a
solution a few years ago:
https://blog.twitter.com/engineering/en_us/a/2013/forward-secrecy-at-twitter.html
This migrates a feature that was previously reserved for enterprise
users, according to https://github.com/caddyserver/caddy/issues/2786.
The PEM loader allows you to embed PEM files (certificates and keys)
directly into your config, rather than requiring them to be stored on
potentially insecure storage, which adds attack vectors. This is useful
in automated settings where sensitive key material is stored only in
memory.
Note that if the config is persisted to disk, that added benefit may go
away, but there will still be the benefit of having lesser dependence on
external files.
This migrates a feature that was previously reserved for enterprise
users, according to https://github.com/caddyserver/caddy/issues/2786.
The local circuit breaker is a simple metrics counter that can cause
the reverse proxy to consider a backend unhealthy before it actually
goes offline, by measuring recent latencies over a sliding window.
Credit to Danny Navarro
This migrates a feature that was previously reserved for enterprise
users, according to https://github.com/caddyserver/caddy/issues/2786.
The cache HTTP handler will be a high-performing, distributed cache
layer for HTTP requests. Right now, the implementation is a very basic
proof-of-concept, and further development is required.
This integrates a feature that was previously reserved for enterprise
users, according to https://github.com/caddyserver/caddy/issues/2786.
The /config and /id endpoints make granular config changes possible as
well as the exporting of the current configuration.
The /load endpoint has been modified to wrap the /config handler so that
the currently-running config can always be available for export. The
difference is that /load allows configs of varying formats and converts
them using config adapters. The adapted config is then processed with
/config as JSON. The /config and /id endpoints accept only JSON.
* v2: introduce CI for v2 branch
* v2-ci: split test report generation from test pass to preserve exit code
* v2-ci: spilt lint results from unit test results
* v2-ci: fix testRunTitle name
* v2-ci: break up the steps for more accurate status indicators
* v2-ci: break steps into different jobs
* v2-ci: revert back to single-job pattern
* v2-ci: reflect the true result by coercing SucceededWithIssues into Failed in the last step
* v2-ci: don't fail the build on lint errors
* cli: Change command structure, add help subcommand (#328)
* cli: improve subcommand structure
- make help command as normal subcommand
- add flag usage message for each command
* cmd: Refactor subcommands and command line help; make commands pluggable
Modules that return an error during provisioning should still be cleaned
up so that they don't leak any resources they may have allocated before
the error occurred. Cleanup should be able to run even if Provision does
not complete fully.
Making them pointers makes for cleaner JSON when adapting configs, if
the struct is empty now it will be omitted entirely.
The x/time/rate package was updated to support changing the burst, so
we've incorporated that here and removed a TODO.
Newlines (\n) can be escaped outside of quoted areas and the newline
will be treated as whitespace but not as an actual line break. Escaping
newlines inside a quoted area is not necessary, and because quotes
trigger literal interpretation of the contents, the escaping backslash
will be parsed as a literal backslash, and the newline will not be
escaped.
Caveat: When a newline is escaped, tokens after it until an unescaped
newline will appear to the parser be on the same line as the initial
token after the last unescaped newline. This may technically lead to
some false line numbers if errors are given, but escaped newlines are
counted so that the next token after an unescaped newline is correct.
See #2766
Before this change, only response headers could be manipulated with the
Caddyfile's 'header' directive.
Also handle the request Host header specially, since the Go standard
library treats it separately from the other header fields...
* Begin WIP integration of HTTP/3 support
* http3: Set actual Handler, make fakeClosePacketConn type for UDP sockets
Also use latest quic-go for ALPN fix
* Manually keep track of and close HTTP/3 listeners
* Update quic-go after working through some http3 bugs
* Fix go mod
* Make http3 optional for now
See: https://stackoverflow.com/a/12518877/1048862
For example, trying to check the existence of "/www/index.php/index.php"
fails but not with an os.IsNotExist()-type error. So we have to assume
that a file that cannot be successfully stat'ed at all does not exist.
- Rename http.var.* -> http.vars.* to be more consistent
- Prefixing a path matcher with * now invokes simple suffix matching
- Handlers and matchers that need a root path default to {http.vars.root}
- Clean replacer output on the file matcher's file selection suffix
* Add support for client TLS authentication
Signed-off-by: Alexandre Stein <alexandre_stein@interlab-net.com>
* make and use client authentication struct
* force StrictSNIHost if TLSConnPolicies is not empty
* Implement leafs verification
* Fixes issue when using multiple verification
* applies the comments from maintainers
* Apply comment
* Refactor/cleanup initial TLS client auth implementation
Use piles from which to draw config values.
Module values can return their name, so now we can do two-way mapping
from value to name and name to value; whereas before we could only map
name to value. This was problematic with the Caddyfile adapter since
it receives values and needs to know the name to put in the config.
./caddy.go:230:12: cannot use *dep (type debug.Module) as type *debug.Module in return argument
./caddy.go:233:12: cannot use bi.Main (type debug.Module) as type *debug.Module in return argument
Along with several other changes, such as renaming caddyhttp.ServerRoute
to caddyhttp.Route, exporting some types that were not exported before,
and tweaking the caddytls TLS values to be more consistent.
Notably, we also now disable automatic cert management for names which
already have a cert (manually) loaded into the cache. These names no
longer need to be specified in the "skip_certificates" field of the
automatic HTTPS config, because they will be skipped automatically.
* optimized functions for inlining
* added note regarding ResponseWriterWrapper
* optimzed browseWrite* methods for FileServer
* created benchmarks for comparison
* creating browseListing instance in each function
* created benchmarks for openResponseWriter
* removed benchmarks of old implementations
* implemented sync.Pool for byte buffers
* using global sync.Pool for writing JSON/HTML
* Tests for Replacer: Replacer.Set and Replacer.Delete
* update replacer test to new implementation
* fix replacer: counted position wrong if placeholder was found
* fix replacer: found placeholder again, if it was a non-existing one
* test with spaces between the placeholders as this could have a different behaviour
* Tests for Replacer.Map
* Tests for Replacer.Set: check also for something like {l{test1}
This should be replaced as {lTEST1REPLACEMENT
* fix replacer: fix multiple occurrence of phOpen sign
* Tests for Replacer: rewrite Set and ReplaceAll tests to use implementation not interface
* Tests for Replacer: rewrite Delete test to use implementation not interface
* Tests for Replacer: rewrite Map tests to use implementation not interface
* Tests for Replacer: add test for NewReplacer
* Tests for Replacer: add test for default replacements
* Tests for Replacer: fixed and refactored tests
* Tests for Replacer: moved default replacement tests to New-test
as new should return a replace with provider which defines global replacements
* Add stop command to admin. Exit after stop.
* Return error on incorrect http Method and provide better logging.
* reuse stopAndCleanup function for all graceful stops
* Force quit /f on windows, also check for processname '.exe' on windows.
* Remove unneeded spaces
* fix tabs
* go fmt tabs
* Return consistent appname which always includes .exe
* Change func name
Differentiating middleware and responders has one benefit, namely that
it's clear which module provides the response, but even then it's not
a great advantage. Linear handler config makes a little more sense,
giving greater flexibility and simplifying the core a bit, even though
it's slightly awkward that handlers which are responders may not use
the 'next' handler that is passed in at all.
Differentiating middleware and responders has one benefit, namely that
it's clear which module provides the response, but even then it's not
a great advantage. Linear handler config makes a little more sense,
giving greater flexibility and simplifying the core a bit, even though
it's slightly awkward that handlers which are responders may not use
the 'next' handler that is passed in at all.
- Fix static responder so it doesn't replace its own headers config,
and instead replaces the actual response header values
- caddyhttp.ResponseRecorder type optionally buffers response
- Add interface guards to ensure regexp matchers get provisioned
- Use default HTTP port if one is not explicitly set
- Encode middleware writes status code 200 if not written upstream
- Templates and markdown only try to execute on text responses
- Static file server sets Content-Type based on file extension only
(this whole thing -- MIME sniffing, etc -- needs more configurability)
This makes it faster and easier to detect broken configurations, but
is a slight performance hit on config loads since we have to re-encode
the decoded struct back into JSON without the module name's key
* set automatic https error type for cert-magic failures
* add state to onload and unload methods
* update reverse proxy to use Provision() and Cleanup()
* Added matcher to determine what protocol the request is being made by
- grpc, tls, http
* Added ability to run caddyscript in a matcher to evaluate the http request
* Added TLS field to caddyscript request time
* Added a library to manipulate and compare a new caddyscript time type
* Library for regex in starlark
Tested for memory leaks and performance. Obviously the added locking and
global state is not awesome, but the alternative is a little uglier IMO:
we'd have to make some sort of "liaison" value which stores the state,
then pass it around to every module, and so LoadModule becomes a lot
less accessible, and each module would need to maintain a reference to
it... nope, just ugly. I think this is the cleaner solution: just make
sure only one Start() happens at a time, and keep global things global.
Very simple log middleware is an example.
Might need to reorder the operations in Start() and handle errors
differently, etc. Otherwise, I'm mostly happy with this solution...
You can have a direct impact on the project by helping with its code. To contribute code to Caddy, open a [pull request](https://github.com/mholt/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/mholt/caddy/-/search).
You can have a huge impact on the project by helping with its code. To contribute code to Caddy, first submit or comment in an issue to discuss your contribution, then open a [pull request](https://github.com/caddyserver/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/caddyserver/caddy).
We hold contributions to a high standard for quality :bowtie:, so don't be surprised if we ask for revisions—even if it seems small or insignificant. Please don't take it personally. :wink: If your change is on the right track, we can guide you to make it mergable.
We hold contributions to a high standard for quality :bowtie:, so don't be surprised if we ask for revisions—even if it seems small or insignificant. Please don't take it personally. :blue_heart: If your change is on the right track, we can guide you to make it mergable.
Here are some of the expectations we have of contributors:
-If your change is more than just a minor alteration, **open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that changes are in-line with the project's goals and the best interests of its users. If there's already an issue about it, comment on the existing issue to claim it.
- **Open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that any changes are in-line with the project's goals and the best interests of its users. We can also discuss the best possible implementation. If there's already an issue about it, comment on the existing issue to claim it. A lot of valuable time can be saved by discussing a proposal first.
- **Keep pull requests small.** Smaller PRs are more likely to be merged because they are easier to review! We might ask you to break up large PRs into smaller ones. [An example of what we DON'T do.](https://twitter.com/iamdevloper/status/397664295875805184)
- **Keep pull requests small.** Smaller PRs are more likely to be merged because they are easier to review! We might ask you to break up large PRs into smaller ones. [An example of what we want to avoid.](https://twitter.com/iamdevloper/status/397664295875805184)
- **Keep related commits together in a PR.** We do want pull requests to be small, but you should also keep multiple related commits in the same PR if they rely on each other.
@@ -45,33 +45,35 @@ Here are some of the expectations we have of contributors:
- **Use comments properly.** We expect good godoc comments for package-level functions, types, and values. Comments are also useful whenever the purpose for a line of code is not obvious.
- **Pull requests may still get closed.** The longer a PR stays open and idle, the more likely it is to be closed. If we haven't reviewed it in a while, it probably means the change is not a priority. Please don't take this personally, we're trying to balance a lot of tasks! If nobody else has commented or reacted to the PR, it likely means your change is useful only to you. The reality is this happens quite a bit. We don't tend to accept PRs that aren't generally helpful. For these reasons or others, the PR may get closed even after a review. We are not obligated to accept all proposed changes, even if the best justification we can give is something vague like, "It doesn't sit right." Sometimes PRs are just the wrong thing or the wrong time. Because it is open source, you can always build your own modified version of Caddy with a change you need, even if we reject it in the official repo.
We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base!
#### HOW TO MAKE A PULL REQUEST TO CADDY
Contributing to Go projects on GitHub is fun and easy. We recommend the following workflow:
Contributing to Go projects on GitHub is fun and easy. After you have proposed your change in an issue, we recommend the following workflow:
1. [Fork this repo](https://github.com/mholt/caddy). This makes a copy of the code you can write to.
1. [Fork this repo](https://github.com/caddyserver/caddy). This makes a copy of the code you can write to.
2. If you don't already have this repo (mholt/caddy.git) repo on your computer, get it with `go get github.com/mholt/caddy/caddy`.
2. If you don't already have this repo (caddyserver/caddy.git) repo on your computer, clone it down: `git clone https://github.com/caddyserver/caddy.git`
3. Tell git that it can push the mholt/caddy.git repo to your fork by adding a remote: `git remote add myfork https://github.com/you/caddy.git`
3. Tell git that it can push the caddyserver/caddy.git repo to your fork by adding a remote: `git remote add myfork https://github.com/<your-username>/caddy.git`
4. Make your changes in the mholt/caddy.git repo on your computer.
4. Make your changes in the caddyserver/caddy.git repo on your computer.
5. Push your changes to your fork: `git push myfork`
6. [Create a pull request](https://github.com/mholt/caddy/pull/new/master) to merge your changes into mholt/caddy @ master. (Click "compare across forks" and change the head fork.)
6. [Create a pull request](https://github.com/caddyserver/caddy/pull/new/master) to merge your changes into caddyserver/caddy @ master. (Click "compare across forks" and change the head fork.)
This workflow is nice because you don't have to change import paths. You can get fancier by using different branches if you want.
### Writing a plugin
### Writing a Caddy module
Caddy can do more with plugins! Anyone can write a plugin. Plugins are Go libraries that get compiled into Caddy, extending its feature set. They can add directives to the Caddyfile, change how the Caddyfile is loaded, and even implement new server types (e.g. HTTP, DNS). When it's ready, you can submit your plugin to the Caddy website so others can download it.
Caddy can do more with modules! Anyone can write one. Caddy modules are Go libraries that get compiled into Caddy, extending its feature set. They can add directives to the Caddyfile, add new configuration adapters, and even implement new server types (e.g. HTTP, DNS).
[Learn how to write and submit a plugin](https://github.com/mholt/caddy/wiki) on the wiki. You should also share and discuss your plugin idea [on the forums](https://caddy.community) to have people test it out. We don't use the Caddy issue tracker for plugins.
[Learn how to write a module here](https://caddyserver.com/docs/extending-caddy). You should also share and discuss your module idea [on the forums](https://caddy.community) to have people test it out. We don't use the Caddy issue tracker for third-party modules.
### Getting help using Caddy
@@ -83,35 +85,61 @@ Many people on the forums could benefit from your experience and expertise, too.
### Reporting bugs
Like every software, Caddy has its flaws. If you find one, [search the issues](https://github.com/mholt/caddy/issues) to see if it has already been reported. If not, [open a new issue](https://github.com/mholt/caddy/issues/new) and describe the bug, and somebody will look into it! (This repository is only for Caddy, not plugins.)
Like every software, Caddy has its flaws. If you find one, [search the issues](https://github.com/caddyserver/caddy/issues) to see if it has already been reported. If not, [open a new issue](https://github.com/caddyserver/caddy/issues/new) and describe the bug, and somebody will look into it! (This repository is only for Caddy and its standard modules.)
**You can help stop bugs in their tracks!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
**You can help us fix bugs!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
Please follow the issue template so we have all the needed information. Unredacted—yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. The burden is on you to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
We may reply with an issue template. Please follow the template so we have all the needed information. Unredacted—yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. If you don't, we might close your report. The burden is on you to make it easily reproducible and to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
We suggest reading [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).
Please be kind. :smile: Remember that Caddy comes at no cost to you, and you're getting free support when we fix your issues. If we helped you, please consider helping someone else!
#### Bug reporting expectations
Maintainers---or more generally, developers---need three things to act on bugs:
1. To agree or be convinced that it's a bug (reporter's responsibility).
- A bug is unintentional, undesired, or surprising behavior which violates documentation or relevant spec. It might be either a mistake in the documentation or a bug in the code.
- This project usually does not work around bugs in other software, systems, and dependencies; instead, we recommend that those bugs are fixed at their source. This sometimes means we close issues or reject PRs that attempt to fix, workaround, or hide bugs in other projects.
2. To be able to understand what is happening (mostly reporter's responsibility).
- If the reporter can provide satisfactory instructions such that a developer can reproduce the bug, the developer will likely be able to understand the bug, write a test case, and implement a fix. This is the least amount of work for everyone and path to the fastest resolution.
- Otherwise, the burden is on the reporter to test possible solutions. This is less preferable because it loosens the feedback loop, slows down debugging efforts, obscures the true nature of the problem from the developers, and is unlikely to result in new test cases.
3. A solution, or ideas toward a solution (mostly maintainer's responsibility).
- Sometimes the best solution is a documentation change.
- Usually the developers have the best domain knowledge for inventing a solution, but reporters may have ideas or preferences for how they would like the software to work.
- Security, correctness, and project goals/vision all take priority over a user's preferences.
- It's simply good business to yield a solution that satisfies the users, and it's even better business to leave them impressed.
Thus, at the very least, the reporter is expected to:
1. Convince the reader that it's a bug in Caddy (if it's not obvious).
2. Reduce the problem down to the minimum specific steps required to reproduce it.
The maintainer is usually able to do the rest; but of course the reporter may invest additional effort to speed up the process.
### Suggesting features
First, [search to see if your feature has already been requested](https://github.com/mholt/caddy/issues). If it has, you can add a :+1: reaction to vote for it. If your feature idea is new, open an issue to request the feature. You don't have to follow the bug template for feature requests. Please describe your idea thoroughly so that we know how to implement it! Really vague requests may not be helpful or actionable and without clarification will have to be closed.
First, [search to see if your feature has already been requested](https://github.com/caddyserver/caddy/issues). If it has, you can add a :+1: reaction to vote for it. If your feature idea is new, open an issue to request the feature. Please describe your idea thoroughly so that we know how to implement it! Really vague requests may not be helpful or actionable and, without clarification, will have to be closed.
While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good plugins](https://github.com/mholt/caddy/wiki), though, which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core.
While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good modules](#writing-a-caddy-module), which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core. Additionally, some features are bad ideas altogether (for either obvious or non-obvious reasons) which may be rejected. We'll try to explain why we reject a feature, but sometimes the best we can do is, "It's not a good fit for the project."
### Improving documentation
Caddy's documentation is available at [https://caddyserver.com/docs](https://caddyserver.com/docs). If you would like to make a fix to the docs, feel free to contribute at the [caddyserver/website](https://github.com/caddyserver/website) repository!
Note that plugin documentation is not hosted by the Caddy website, other than basic usage examples. They are managed by the individual plugin authors, and you will have to contact them to change their documentation.
Caddy's documentation is available at [https://caddyserver.com/docs](https://caddyserver.com/docs) and its source is in the [website repo](https://github.com/caddyserver/website). If you would like to make a fix to the docs, please submit an issue there describing the change to make.
Note that third-party module documentation is not hosted by the Caddy website, other than basic usage examples. They are managed by the individual module authors, and you will have to contact them to change their documentation.
Our documentation is scoped to the Caddy project only: it is not for describing how other software or systems work, even if they relate to Caddy or web servers. That kind of content [can be found in our community wiki](https://caddy.community/c/wiki/13), however.
## Collaborator Instructions
Collabators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help.The expectations we have of collaborators are:
Collaborators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help.The expectations we have of collaborators are:
- **Help review pull requests.** Be meticulous, but also kind. We love our contributors, but we critique the contribution to make it better. Multiple, thorough reviews make for the best contributions! Here are some questions to consider:
- Can the change be made more elegant?
@@ -128,9 +156,9 @@ Collabators have push rights to the repository. We grant this permission after o
- **Prefer squashed commits over a messy merge.** If there are many little commits, please [squash the commits](https://stackoverflow.com/a/11732910/1048862) so we don't clutter the commit history.
- **Don't accept new dependencies lightly.** Dependencies can make the world crash and burn, but they are sometimes necessary. Choose carefully. Extremely small dependencies (a few lines of code) can be inlined. The rest may not be needed. For those that are, Caddy vendors all dependencies with the help of [gvt](https://github.com/FiloSottile/gvt). All external dependencies must be vendored, and _Caddy must not export any types defined by those dependencies_. Check this diligently!
- **Don't accept new dependencies lightly.** Dependencies can make the world crash and burn, but they are sometimes necessary. Choose carefully. Extremely small dependencies (a few lines of code) can be inlined. The rest may not be needed. For those that are, Caddy uses [go modules](https://github.com/golang/go/wiki/Modules). All external dependencies must be installed as modules, and _Caddy must not export any types defined by those dependencies_. Check this diligently!
- **Be extra careful in some areas of the code.** There are some critical areas in the Caddy code base that we review extra meticulously: the `caddy` and `caddytls` packages especially.
- **Be extra careful in some areas of the code.** There are some critical areas in the Caddy code base that we review extra meticulously: the `caddyhttp` and `caddytls` packages especially.
- **Make sure tests test the actual thing.** Double-check that the tests fail without the change, and pass with it. It's important that they assert what they're purported to assert.
@@ -142,19 +170,18 @@ Collabators have push rights to the repository. We grant this permission after o
## Values
## Values (WIP)
- A person is always more important than code. People don't like being handled "efficiently". But we can still process issues and pull requests efficiently while being kind, patient, and considerate.
- The ends justify the means, if the means are good. A good tree won't produce bad fruit. But if we cut corners or are hasty in our process, the end result will not be good.
## Responsible Disclosure
## Security Policy
If you've found a security vulnerability, please email me, the author, directly: Matthew dot Holt at Gmail. I'll need enough information to verify the bug and make a patch. It will speed things up if you suggest a working patch. If your report is valid and a patch is released, we will not reveal your identity by default. If you wish to be credited, please give me the name to use. Thanks for responsibly helping Caddy—and thousands of websites—be more secure!
If you think you've found a security vulnerability, please refer to our [Security Policy](https://github.com/caddyserver/caddy/security/policy) document.
## Thank you
Thanks for your help! Caddy would not be what it is today without your
contributions.
Thanks for your help! Caddy would not be what it is today without your contributions.
Are you asking for help with using Caddy? Please use our forum instead: https://caddy.community. If you are filing a bug report, please take a few minutes to carefully answer the following questions. If your issue is not a bug report, you do not need to use this template. Thanks!)
-->
### 1. What version of Caddy are you using (`caddy -version`)?
### 2. What are you trying to do?
### 3. What is your entire Caddyfile?
```text
(paste Caddyfile here)
```
### 4. How did you run Caddy (give the full command and describe the execution environment)?
### 5. Please paste any relevant HTTP request(s) here.
<!-- Paste curl command, or full HTTP request including headers and body, here. -->
### 6. What did you expect to see?
### 7. What did you see instead (give full error messages and/or log)?
### 8. How can someone who is starting from scratch reproduce the bug as minimally as possible?
<!-- Please strip away any extra infrastructure such as containers, reverse proxies, upstream apps, caches, dependencies, etc, to prove this is a bug in Caddy and not an external misconfiguration. Your chances of getting this bug fixed go way up the easier it is to replicate. Thank you! -->
The Caddy project would like to make sure that it stays on top of all practically-exploitable vulnerabilities.
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| 2.x | :white_check_mark: |
| 1.x | :x: |
| < 1.x | :x: |
## Acceptable Scope
A security report must demonstrate a security bug in the source code from this repository.
Some security problems are the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please only report vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing, BGP hijacks, or missing/misconfigured HTTP headers a vulnerability in the Caddy web server).
Vulnerabilities caused by misconfigurations are out of scope. Yes, it is entirely possible to craft and use a configuration that is unsafe, just like with every other web server; we recommend against doing that.
We do not accept reports if the steps imply or require a compromised system or third-party software, as we cannot control those. We expect that users secure their own systems and keep all their software patched. For example, if untrusted users are able to upload/write/host arbitrary files in the web root directory, it is NOT a security bug in Caddy if those files get served to clients; however, it _would_ be a valid report if a bug in Caddy's source code unintentionally gave unauthorized users the ability to upload unsafe files or delete files without relying on an unpatched system or piece of software.
Client-side exploits are out of scope. In other words, it is not a bug in Caddy if the web browser does something unsafe, even if the downloaded content was served by Caddy. (Those kinds of exploits can generally be mitigated by proper configuration of HTTP headers.) As a general rule, the content served by Caddy is not considered in scope because content is configurable by the site owner or the associated web application.
Security bugs in code dependencies are out of scope. Instead, if a dependency has patched a relevant security bug, please feel free to open a public issue or pull request to update that dependency in our code.
## Reporting a Vulnerability
We get a lot of difficult reports that turn out to be invalid. Clear, obvious reports tend to be the most credible (but are also rare).
First please ensure your report falls within the accepted scope of security bugs (above).
We'll need enough information to verify the bug and make a patch. To speed things up, please include:
- Most minimal possible config (without redactions!)
- Command(s)
- Precise HTTP requests (`curl -v` and its output please)
- Full log output (please enable debug mode)
- Specific minimal steps to reproduce the issue from scratch
- A working patch
Please DO NOT use containers, VMs, cloud instances or services, or any other complex infrastructure in your steps. Always prefer `curl` instead of web browsers.
We consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems. Do not redact domain names from your reports. The actual content of your domain name affects Caddy's behavior, so we need the exact domain name(s) to reproduce with, or your report will be ignored.
It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, depending on available resources. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. (We get a lot of invalid reports.) Thank you for understanding.
When you are ready, please email Matt Holt (the author) directly: matt at dyanim dot com.
Please don't encrypt the email body. It only makes the process more complicated.
Please also understand that due to our nature as an open source project, we do not have a budget to award security bounties. We can only thank you.
If your report is valid and a patch is released, we will not reveal your identity by default. If you wish to be credited, please give us the name to use and/or your GitHub username. If you don't provide this we can't credit you.
Thanks for responsibly helping Caddy—and thousands of websites—be more secure!
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -t caddy-ci@ci-s390x.caddyserver.com "cd /var/tmp/$short_sha; go version; go env; printf "\n\n";CGO_ENABLED=0 go test -v ./..."
<a href="https://twitter.com/caddyserver" title="@caddyserver on Twitter"><img src="https://img.shields.io/badge/twitter-@caddyserver-55acee.svg" alt="@caddyserver on Twitter"></a>
<a href="https://sourcegraph.com/github.com/mholt/caddy?badge" title="Caddy on Sourcegraph"><img src="https://sourcegraph.com/github.com/mholt/caddy/-/badge.svg" alt="Caddy on Sourcegraph"></a>
<br>
<a href="https://sourcegraph.com/github.com/caddyserver/caddy?badge" title="Caddy on Sourcegraph"><img src="https://sourcegraph.com/github.com/caddyserver/caddy/-/badge.svg" alt="Caddy on Sourcegraph"></a>
- **Automatic HTTPS** on by default (via [Let's Encrypt](https://letsencrypt.org))
- **HTTP/2** by default
- **Virtual hosting** so multiple sites just work
-Experimental **QUIC support** for those that like speed
-TLS session ticket **key rotation** for more secure connections
-**Extensible with plugins** because a convenient web server is a helpful one
## [Features](https://caddyserver.com/v2)
- **Easy configuration** with the [Caddyfile](https://caddyserver.com/docs/caddyfile)
-**Powerful configuration** with its [native JSON config](https://caddyserver.com/docs/json/)
-**Dynamic configuration** with the [JSON API](https://caddyserver.com/docs/api)
-[**Config adapters**](https://caddyserver.com/docs/config-adapters) if you don't like JSON
- **Automatic HTTPS** by default
- [ZeroSSL](https://zerossl.com) and [Let's Encrypt](https://letsencrypt.org) for public names
- Fully-managed local CA for internal names & IPs
- Can coordinate with other Caddy instances in a cluster
- Multi-issuer fallback
- **Stays up when other servers go down** due to TLS/OCSP/certificate-related issues
- **Production-ready** after serving trillions of requests and managing millions of TLS certificates
- **Scales to hundreds of thousands of sites** as proven in production
- **HTTP/1.1, HTTP/2, and HTTP/3** supported all by default
- **Highly extensible** [modular architecture](https://caddyserver.com/docs/architecture) lets Caddy do anything without bloat
- **Runs anywhere** with **no external dependencies** (not even libc)
There's way more, too! [See all features built into Caddy.](https://caddyserver.com/features) On top of all those, Caddy does even more with plugins: choose which plugins you want at [download](https://caddyserver.com/download).
- Written in Go, a language with higher **memory safety guarantees** than other servers
- Actually **fun to use**
- So much more to [discover](https://caddyserver.com/v2)
## Install
Caddy binaries have no dependencies and are available for every platform. Get Caddy any one of these ways:
The simplest, cross-platform way to get started is to download Caddy from [GitHub Releases](https://github.com/caddyserver/caddy/releases) and place the executable file in your PATH.
- **[Download page](https://caddyserver.com/download)** allows you to
customize your build in the browser
- **[Latest release](https://github.com/mholt/caddy/releases/latest)** for
pre-built, vanilla binaries
- **go get** to build from source: `go get github.com/mholt/caddy/caddy` (requires Go 1.8 or newer)
See [our online documentation](https://caddyserver.com/docs/install) for other install instructions.
Then make sure the `caddy` binary is in your PATH.
## Build from source
Requirements:
## Quick Start
- [Go 1.18 or newer](https://golang.org/dl/)
To serve static files from the current working directory, run:
### For development
_**Note:** These steps [will not embed proper version information](https://github.com/golang/go/issues/29228). For that, please follow the instructions in the next section._
Caddy's default port is 2015, so open your browser to [http://localhost:2015](http://localhost:2015).
When you run Caddy, it may try to bind to low ports unless otherwise specified in your config. If your OS requires elevated privileges for this, you will need to give your new binary permission to do so. On Linux, this can be done easily with: `sudo setcap cap_net_bind_service=+ep ./caddy`
### Go from 0 to HTTPS in 5 seconds
If you prefer to use `go run` which only creates temporary binaries, you can still do this with the included `setcap.sh` like so:
If the `caddy` binary has permission to bind to low ports and your domain name's DNS records point to the machine you're on:
```
caddy -host example.com
```bash
$ go run -exec ./setcap.sh main.go
```
This command serves static files from the current directory over HTTPS. Certificates are automatically obtained and renewed for you!
If you don't want to type your password for `setcap`, use `sudo visudo` to edit your sudoers file and allow your user account to run that command without a password, for example:
### Customizing your site
To customize how your site is served, create a file named Caddyfile by your site and paste this into it:
```plain
localhost
push
browse
websocket /echo cat
ext .html
log /var/log/access.log
proxy /api 127.0.0.1:7005
header /api Access-Control-Allow-Origin *
```
username ALL=(ALL:ALL) NOPASSWD: /usr/sbin/setcap
```
When you run `caddy` in that directory, it will automatically find and use that Caddyfile.
replacing `username` with your actual username. Please be careful and only do this if you know what you are doing! We are only qualified to document how to use Caddy, not Go tooling or your computer, and we are providing these instructions for convenience only; please learn how to use your own computer at your own risk and make any needful adjustments.
This simple file enables server push (via Link headers), allows directory browsing (for folders without an index file), hosts a WebSocket echo server at /echo, serves clean URLs, logs requests to an access log, proxies all API requests to a backend on port 7005, and adds the coveted `Access-Control-Allow-Origin: *` header for all responses from the API.
### With version information and/or plugins
Wow! Caddy can do a lot with just a few lines.
Using [our builder tool, `xcaddy`](https://github.com/caddyserver/xcaddy)...
### Doing more with Caddy
```
$ xcaddy build
```
To host multiple sites and do more with the Caddyfile, please see the [Caddyfile tutorial](https://caddyserver.com/tutorial/caddyfile).
...the following steps are automated:
Sites with qualifying hostnames are served over [HTTPS by default](https://caddyserver.com/docs/automatic-https).
Caddy has a command line interface. Run `caddy -h` to view basic help or see the [CLI documentation](https://caddyserver.com/docs/cli) for details.
1. Create a new folder: `mkdir caddy`
2. Change into it: `cd caddy`
3. Copy [Caddy's main.go](https://github.com/caddyserver/caddy/blob/master/cmd/caddy/main.go) into the empty folder. Add imports for any custom plugins you want to add.
4. Initialize a Go module: `go mod init caddy`
5. (Optional) Pin Caddy version: `go get github.com/caddyserver/caddy/v2@version` replacing `version` with a git tag, commit, or branch name.
6. (Optional) Add plugins by adding their import: `_ "import/path/here"`
7. Compile: `go build`
## Running in Production
Caddy is production-ready if you find it to be a good fit for your site and workflow.
**Running as root:** We advise against this. You can still listen on ports < 1024 on Linux using setcap like so: `sudo setcap cap_net_bind_service=+ep ./caddy`
The Caddy project does not officially maintain any system-specific integrations nor suggest how to administer your own system. But your download file includes [unofficial resources](https://github.com/mholt/caddy/tree/master/dist/init) contributed by the community that you may find helpful for running Caddy in production.
How you choose to run Caddy is up to you. Many users are satisfied with `nohup caddy &`. Others use `screen`. Users who need Caddy to come back up after reboots either do so in the script that caused the reboot, add a command to an init script, or configure a service with their OS.
If you have questions or concerns about Caddy' underlying crypto implementations, consult Go's [crypto packages](https://golang.org/pkg/crypto), starting with their documentation, then issues, then the code itself; as Caddy uses mainly those libraries.
## Contributing
## Quick start
**[Join our forum](https://caddy.community) where you can chat with other Caddy users and developers!** To get familiar with the code base, try [Caddy code search on Sourcegraph](https://sourcegraph.com/github.com/mholt/caddy/-/search)!
The [Caddy website](https://caddyserver.com/docs/) has documentation that includes tutorials, quick-start guides, reference, and more.
Please see our [contributing guidelines](https://github.com/mholt/caddy/blob/master/.github/CONTRIBUTING.md) for instructions. If you want to write a plugin, check out the [developer wiki](https://github.com/mholt/caddy/wiki).
**We recommend that all users -- regardless of experience level -- do our [Getting Started](https://caddyserver.com/docs/getting-started) guide to become familiar with using Caddy.**
We use GitHub issues and pull requests only for discussing bug reports and the development of specific changes. We welcome all other topics on the [forum](https://caddy.community)!
If you want to contribute to the documentation, please submit pull requests to [caddyserver/website](https://github.com/caddyserver/website).
Thanks for making Caddy -- and the Web -- better!
If you've only got a minute, [the website has several quick-start tutorials](https://caddyserver.com/docs/quick-starts) to choose from! However, after finishing a quick-start tutorial, please read more documentation to understand how the software works. 🙂
## Donors
- [DigitalOcean](https://m.do.co/c/6d7bdafccf96) is hosting the Caddy project.
- [DNSimple](https://dnsimple.link/resolving-caddy) provides DNS services for Caddy's sites.
- [DNS Spy](https://dnsspy.io) keeps an eye on Caddy's DNS properties.
We thank them for their services. **If you want to help keep Caddy free, please [become a sponsor](https://caddyserver.com/pricing)!**
## About the Project
## Overview
Caddy was born out of the need for a "batteries-included" web server that runs anywhere and doesn't have to take its configuration with it. Caddy took inspiration from [spark](https://github.com/rif/spark), [nginx](https://github.com/nginx/nginx), lighttpd,
[Websocketd](https://github.com/joewalnes/websocketd) and [Vagrant](https://www.vagrantup.com/), which provides a pleasant mixture of features from each of them.
Caddy is most often used as an HTTPS server, but it is suitable for any long-running Go program. First and foremost, it is a platform to run Go applications. Caddy "apps" are just Go programs that are implemented as Caddy modules. Two apps -- `tls` and `http` -- ship standard with Caddy.
**The name "Caddy":** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". See [brand guidelines](https://caddyserver.com/brand).
Caddy apps instantly benefit from [automated documentation](https://caddyserver.com/docs/json/), graceful on-line [config changes via API](https://caddyserver.com/docs/api), and unification with other Caddy apps.
*Author on Twitter: [@mholt6](https://twitter.com/mholt6)*
Although [JSON](https://caddyserver.com/docs/json/) is Caddy's native config language, Caddy can accept input from [config adapters](https://caddyserver.com/docs/config-adapters) which can essentially convert any config format of your choice into JSON: Caddyfile, JSON 5, YAML, TOML, NGINX config, and more.
The primary way to configure Caddy is through [its API](https://caddyserver.com/docs/api), but if you prefer config files, the [command-line interface](https://caddyserver.com/docs/command-line) supports those too.
Caddy exposes an unprecedented level of control compared to any web server in existence. In Caddy, you are usually setting the actual values of the initialized types in memory that power everything from your HTTP handlers and TLS handshakes to your storage medium. Caddy is also ridiculously extensible, with a powerful plugin system that makes vast improvements over other web servers.
To wield the power of this design, you need to know how the config document is structured. Please see [our documentation site](https://caddyserver.com/docs/) for details about [Caddy's config structure](https://caddyserver.com/docs/json/).
Nearly all of Caddy's configuration is contained in a single config document, rather than being scattered across CLI flags and env variables and a configuration file as with other web servers. This makes managing your server config more straightforward and reduces hidden variables/factors.
## Full documentation
Our website has complete documentation:
**https://caddyserver.com/docs/**
The docs are also open source. You can contribute to them here: https://github.com/caddyserver/website
## Getting help
- We advise companies using Caddy to secure a support contract through [Ardan Labs](https://www.ardanlabs.com/my/contact-us?dd=caddy) before help is needed.
- A [sponsorship](https://github.com/sponsors/mholt) goes a long way! We can offer private help to sponsors. If Caddy is benefitting your company, please consider a sponsorship. This not only helps fund full-time work to ensure the longevity of the project, it provides your company the resources, support, and discounts you need; along with being a great look for your company to your customers and potential customers!
- Individuals can exchange help for free on our community forum at https://caddy.community. Remember that people give help out of their spare time and good will. The best way to get help is to give it first!
Please use our [issue tracker](https://github.com/caddyserver/caddy/issues) only for bug reports and feature requests, i.e. actionable development items (support questions will usually be referred to the forums).
## About
Matthew Holt began developing Caddy in 2014 while studying computer science at Brigham Young University. (The name "Caddy" was chosen because this software helps with the tedious, mundane tasks of serving the Web, and is also a single place for multiple things to be organized together.) It soon became the first web server to use HTTPS automatically and by default, and now has hundreds of contributors and has served trillions of HTTPS requests.
**The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of Stack Holdings GmbH.
- _Project on Twitter: [@caddyserver](https://twitter.com/caddyserver)_
- _Author on Twitter: [@mholt6](https://twitter.com/mholt6)_
Caddy is a project of [ZeroSSL](https://zerossl.com), a Stack Holdings company.
Debian package repository hosting is graciously provided by [Cloudsmith](https://cloudsmith.com). Cloudsmith is the only fully hosted, cloud-native, universal package management solution, that enables your organization to create, store and share packages in any format, to any place, with total confidence.
// directives which return a literal subroute instead of a route
// means they intend to keep those handlers together without
// them being reordered; we're doing that anyway since we're in
// the route directive, so just append its handlers
sr.Routes=append(sr.Routes,handler.Routes...)
default:
returnnil,h.Errf("%s directive returned something other than an HTTP route or subroute: %#v (only handler directives can be used in routes)",result.directive,result.Value)
caddy.Log().Named("caddyfile").Warn("DEPRECATED: protocol > strict_sni_host in this position will be removed soon; move up to the servers block instead")
returnnil,warnings,fmt.Errorf("automation policy from site block is also default/catch-all policy because of key without hostname, and the two are in conflict: %#v != %#v",ap.Issuers,issuers)
// first make sure this block is allowed to create an automation policy;
// doing so is forbidden if it has a key with no host (i.e. ":443")
// and if there is a different server block that also has a key with no
// host -- since a key with no host matches any host, we need its
// associated automation policy to have an empty Subjects list, i.e. no
// host filter, which is indistinguishable between the two server blocks
// because automation is not done in the context of a particular server...
// this is an example of a poor mapping from Caddyfile to JSON but that's
// the least-leaky abstraction I could figure out
iflen(sblockHosts)==0{
ifserverBlocksWithTLSHostlessKey>1{
// this server block and at least one other has a key with no host,
// making the two indistinguishable; it is misleading to define such
// a policy within one server block since it actually will apply to
// others as well
returnnil,warnings,fmt.Errorf("cannot make a TLS automation policy from a server block that has a host-less address when there are other TLS-enabled server block addresses lacking a host")
}
ifcatchAllAP==nil{
// this server block has a key with no hosts, but there is not yet
// a catch-all automation policy (probably because no global options
// were set), so this one becomes it
catchAllAP=ap
}
}
// associate our new automation policy with this server block's hosts
ap.Subjects=sblock.hostsFromKeysNotHTTP(httpPort)
sort.Strings(ap.Subjects)// solely for deterministic test results
// if a combination of public and internal names were given
// for this same server block and no issuer was specified, we
// need to separate them out in the automation policies so
// that the internal names can use the internal issuer and
// the other names can use the default/public/ACME issuer
varap2*caddytls.AutomationPolicy
iflen(ap.Issuers)==0{
varinternal,external[]string
for_,s:=rangeap.Subjects{
if!certmagic.SubjectQualifiesForCert(s){
returnnil,warnings,fmt.Errorf("subject does not qualify for certificate: '%s'",s)
}
// we don't use certmagic.SubjectQualifiesForPublicCert() because of one nuance:
// names like *.*.tld that may not qualify for a public certificate are actually
// fine when used with OnDemand, since OnDemand (currently) does not obtain
// wildcards (if it ever does, there will be a separate config option to enable
// it that we would need to check here) since the hostname is known at handshake;
// and it is unexpected to switch to internal issuer when the user wants to get
// regular certificates on-demand for a class of certs like *.*.tld.
// if there are no global options related to automation policies
// set, then we can just return right away
if!hasGlobalAutomationOpts{
ifalways{
returnnew(caddytls.AutomationPolicy),nil
}
returnnil,nil
}
ap:=new(caddytls.AutomationPolicy)
ifhasKeyType{
ap.KeyType=keyType.(string)
}
ifhasIssuers&&hasLocalCerts{
returnnil,fmt.Errorf("global options are ambiguous: local_certs is confusing when combined with cert_issuer, because local_certs is also a specific kind of issuer")
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Matt Holt
Francis Lavoie
Mohammed Al Sahaf
fleandro
Dave Henderson
WeidiDeng
Avdhut
Abirdcfly
Dávid Szabó
Simon Legner
Ben Burkert
jedy
WilczyńskiT
Abdussamet Koçak
Kévin Dunglas
lewandowski-stripe
Chirag Maheshwari
WingLim
Y.Horie
jhwz
Tristan Swadell
Kiss Károly Pál
Kiss Károly Pál
Yaacov Akiba Slama
Gr33nbl00d
Alexander M
Aleks
David Larlet
世界
Tyler Kropp
Marco Kaufmann
cui fliter
Artem Mikheev
Andrii Kushch
Ran Chen
ttys3
BitWuehler
Noorain Panjwani
Alok Naushad
Kevin Daudt
Vojtech Vitek
Forest Johnson
GallopingKylin
rayjlinden
Денис Телюх
Rainer Borene
Runzhi He
Adam Burgess
Kévin Dunglas
Tim Culverhouse
Jeremy Lin
Marc Easen
Klaus Helenius
Simão Gomes Viana
M. Ángel Jimeno
Oleg
KallyDev
Slavik
HayatoShiba
peymaneh
Steffen Brüheim
Pascal Zarrad
Adam Weinberger
Scott Mebberson
Steven Angles
Frederik Ring
王清雨
Ggicci
Leo Di Donato
diamondburned
elnoro
hmol233
mritd
Klaus Post
Klooven
Peter Magnusson
diamondburned
Matthew Penner
Jesse
Calvin Xiao
Jason Du
Alban Lecocq
Simão Gomes Viana
Carl George
Marten Seemann
Dimitri Masson
Simão Gomes Viana
Simão Gomes Viana
rai
Aaron Taylor
Rajat Jain
Tyler Kropp
Sven Dowideit
Stefan Tatschner
go-d
Jordi Masip
yaxin
Jack Baron
Cuong Manh Le
Daniel Santos
Ian
Gilbert Gilb's
Nicola Piccinini
Aurelia
Gaurav Dhameeja
Christoph Kluge
Jason McCallister
AJ ONeal
Aleksei
Christian Flach
Kevin Lin
Ye Zhihao
Bart
v-rosa
Patrick Hein
Manuel Dalla Lana
snu-ceyda
Greg Anders
James Birtles
Mark Sargent
Xiuming Chen
Chris Ortman
Wynn Wolf Arbor
NWHirschfeld
Georges Haidar
Rui Lopes
AndyBan
Thorkild Gregersen
elcore
linquize
Gregory Dosh
Andrew Zhou
Chandler Swift
Jose Donizetti
Karol Będkowski
Daniel Black
Christoffer Andersson
westwin
Robin Lambertz
Pascal
Paolo Barbolini
Vaibhav
Bill Glover
evtr
Success Go
Zaq? Wiedmann
Cameron Moore
Success Go
Robin Lambertz
Jeremy Lin
Abdelmalek Ihdene
lu4p
Sarat Chandra
Andreas Schneider
yzongyue
aca
Ingo Gottwald
Alexandre Stein
Ariel Núñez
Albert Shirima
Dominik Braun
Dreamacro
Johannes Hörmann
Toby Allen
dev
Matt Holt