caddyhttp: Sanitize the path before evaluating path matchers (#4407)

* caddycmd: Add `--skip-cleanup` to upgrade commands

This is a partial fix for https://github.com/caddyserver/caddy/issues/4057, making it possible to retain the old build of Caddy, in case something went wrong.

* caddycmd: Fix duplicate error message

The error message "download succeeded, but unable to execute" was repeated, because it was both in the `listModules`/`showVersion` functions and in the calling `upgradeBuild` function. Oversight when this was refactored.

* caddycmd: Implement fix for performing cleanup on Windows

Without this, the cleanup operation would fail with an error message like this:

upgrade: download succeeded, but unable to clean up backup binary: remove C:\caddy\caddy.exe.tmp: Access is denied.

* caddycmd: Rename to `--keep-backup`, simplify build constraints

.github/SECURITY.md

@@ -2,9 +2,6 @@
 
 The Caddy project would like to make sure that it stays on top of all practically-exploitable vulnerabilities.
 
-Some security problems are more the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please report only vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing or BGP hijacks a vulnerability in the Caddy web server).
-
-Please note that we consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems.
 
 ## Supported Versions
 
@@ -14,11 +11,46 @@ Please note that we consider publicly-registered domain names to be public infor
 | 1.x     | :x:                |
 | < 1.x   | :x:                |
 
+
+## Acceptable Scope
+
+A security report must demonstrate a security bug in the source code from this repository.
+
+Some security problems are the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please only report vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing, BGP hijacks, or missing/misconfigured HTTP headers a vulnerability in the Caddy web server).
+
+Vulnerabilities caused by misconfigurations are out of scope. Yes, it is entirely possible to craft and use a configuration that is unsafe, just like with every other web server; we recommend against doing that.
+
+We do not accept reports if the steps imply or require a compromised system or third-party software, as we cannot control those. We expect that users secure their own systems and keep all their software patched. For example, if untrusted users are able to upload/write/host arbitrary files in the web root directory, it is NOT a security bug in Caddy if those files get served to clients; however, it _would_ be a valid report if a bug in Caddy's source code unintentionally gave unauthorized users the ability to upload unsafe files or delete files without relying on an unpatched system or piece of software.
+
+Client-side exploits are out of scope. In other words, it is not a bug in Caddy if the web browser does something unsafe, even if the downloaded content was served by Caddy. (Those kinds of exploits can generally be mitigated by proper configuration of HTTP headers.) As a general rule, the content served by Caddy is not considered in scope because content is configurable by the site owner or the associated web application.
+
+Security bugs in code dependencies are out of scope. Instead, if a dependency has patched a relevant security bug, please feel free to open a public issue or pull request to update that dependency in our code.
+
+
 ## Reporting a Vulnerability
 
-Please email Matt Holt (the author) directly: matt [at] lightcodelabs [dot com].
+We get a lot of difficult reports that turn out to be invalid. Clear, obvious reports tend to be the most credible (but are also rare).
 
-We'll need enough information to verify the bug and make a patch. It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, resources permitting. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. Thank you for understanding.
+First please ensure your report falls within the accepted scope of security bugs (above).
+
+We'll need enough information to verify the bug and make a patch. To speed things up, please include:
+
+- Most minimal possible config (without redactions!)
+- Command(s)
+- Precise HTTP requests (`curl -v` and its output please)
+- Full log output (please enable debug mode)
+- Specific minimal steps to reproduce the issue from scratch
+- A working patch
+
+Please DO NOT use containers, VMs, cloud instances or services, or any other complex infrastructure in your steps. Always prefer `curl` instead of web browsers.
+
+We consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems. Do not redact domain names from your reports. The actual content of your domain name affects Caddy's behavior, so we need the exact domain name(s) to reproduce with, or your report will be ignored.
+
+It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, depending on available resources. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. (We get a lot of invalid reports.) Thank you for understanding.
+
+When you are ready, please email Matt Holt (the author) directly: matt at dyanim dot com.
+
+Please don't encrypt the email body. It only makes the process more complicated.
 
 Please also understand that due to our nature as an open source project, we do not have a budget to award security bounties. We can only thank you.
 

.github/workflows/ci.yml

@@ -19,7 +19,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ ubuntu-latest, macos-latest, windows-latest ]
-        go: [ '1.15', '1.16' ]
+        go: [ '1.16', '1.17' ]
 
         # Set some variables per OS, usable via ${{ matrix.VAR }}
         # CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
@@ -156,6 +156,7 @@ jobs:
     steps:
       - name: checkout
         uses: actions/checkout@v2
+      
       - uses: goreleaser/goreleaser-action@v2
         with:
           version: latest

.github/workflows/cross-build.yml

@@ -16,7 +16,7 @@ jobs:
       fail-fast: false
       matrix:
         goos: ['android', 'linux', 'solaris', 'illumos', 'dragonfly', 'freebsd', 'openbsd', 'plan9', 'windows', 'darwin', 'netbsd']
-        go: [ '1.15', '1.16' ]
+        go: [ '1.17' ]
     runs-on: ubuntu-latest
     continue-on-error: true
     steps:

.github/workflows/release.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest ]
-        go: [ '1.16' ]
+        go: [ '1.17' ]
     runs-on: ${{ matrix.os }}
 
     steps:

.golangci.yml

@@ -1,6 +1,6 @@
 linters-settings:
   errcheck:
-    ignore: fmt:.*,io/ioutil:^Read.*,go.uber.org/zap/zapcore:^Add.*
+    ignore: fmt:.*,go.uber.org/zap/zapcore:^Add.*
     ignoretests: true
 
 linters:

.goreleaser.yml

@@ -75,11 +75,11 @@ nfpms:
   - id: default
     package_name: caddy
 
-    vendor: Light Code Labs
+    vendor: Dyanim
     homepage: https://caddyserver.com
     maintainer: Matthew Holt <mholt@users.noreply.github.com>
     description: |
-      Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
+      Caddy - Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
     license: Apache 2.0
 
     formats:

README.md

@@ -75,7 +75,7 @@ For other install options, see https://caddyserver.com/docs/install.
 
 Requirements:
 
-- [Go 1.15 or newer](https://golang.org/dl/)
+- [Go 1.16 or newer](https://golang.org/dl/)
 
 ### For development
  
@@ -176,6 +176,8 @@ Please use our [issue tracker](https://github.com/caddyserver/caddy/issues) only
 
 ## About
 
+Matthew Holt began developing Caddy in 2014 while studying computer science at Brigham Young University. (The name "Caddy" was chosen because this software helps with the tedious, mundane tasks of serving the Web, and is also a single place for multiple things to be organized together.) It soon became the first web server to use HTTPS automatically and by default, and now has hundreds of contributors and has served trillions of HTTPS requests.
+
 **The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of Stack Holdings GmbH.
 
 - _Project on Twitter: [@caddyserver](https://twitter.com/caddyserver)_

admin.go

@@ -26,7 +26,6 @@ import (
 	"expvar"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"net"
 	"net/http"
 	"net/http/pprof"
@@ -109,6 +108,12 @@ type ConfigSettings struct {
 	//
 	// EXPERIMENTAL: Subject to change.
 	LoadRaw json.RawMessage `json:"load,omitempty" caddy:"namespace=caddy.config_loaders inline_key=module"`
+
+	// The interval to pull config. With a non-zero value, will pull config
+	// from config loader (eg. a http loader) with given interval.
+	//
+	// EXPERIMENTAL: Subject to change.
+	LoadInterval Duration `json:"load_interval,omitempty"`
 }
 
 // IdentityConfig configures management of this server's identity. An identity
@@ -329,6 +334,7 @@ func replaceLocalAdminServer(cfg *Config) error {
 		return err
 	}
 
+	serverMu.Lock()
 	localAdminServer = &http.Server{
 		Addr:              addr.String(), // for logging purposes only
 		Handler:           handler,
@@ -337,10 +343,14 @@ func replaceLocalAdminServer(cfg *Config) error {
 		IdleTimeout:       60 * time.Second,
 		MaxHeaderBytes:    1024 * 64,
 	}
+	serverMu.Unlock()
 
 	adminLogger := Log().Named("admin")
 	go func() {
-		if err := localAdminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+		serverMu.Lock()
+		server := localAdminServer
+		serverMu.Unlock()
+		if err := server.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
 			adminLogger.Error("admin server shutdown for unknown reason", zap.Error(err))
 		}
 	}()
@@ -364,11 +374,6 @@ func manageIdentity(ctx Context, cfg *Config) error {
 		return nil
 	}
 
-	oldIdentityCertCache := identityCertCache
-	if oldIdentityCertCache != nil {
-		defer oldIdentityCertCache.Stop()
-	}
-
 	// set default issuers; this is pretty hacky because we can't
 	// import the caddytls package -- but it works
 	if cfg.Admin.Identity.IssuersRaw == nil {
@@ -389,8 +394,13 @@ func manageIdentity(ctx Context, cfg *Config) error {
 		}
 	}
 
+	// we'll make a new cache when we make the CertMagic config, so stop any previous cache
+	if identityCertCache != nil {
+		identityCertCache.Stop()
+	}
+
 	logger := Log().Named("admin.identity")
-	cmCfg := cfg.Admin.Identity.certmagicConfig(logger)
+	cmCfg := cfg.Admin.Identity.certmagicConfig(logger, true)
 
 	// issuers have circular dependencies with the configs because,
 	// as explained in the caddytls package, they need access to the
@@ -456,7 +466,7 @@ func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
 	}
 
 	// create TLS config that will enforce mutual authentication
-	cmCfg := cfg.Admin.Identity.certmagicConfig(remoteLogger)
+	cmCfg := cfg.Admin.Identity.certmagicConfig(remoteLogger, false)
 	tlsConfig := cmCfg.TLSConfig()
 	tlsConfig.NextProtos = nil // this server does not solve ACME challenges
 	tlsConfig.ClientAuth = tls.RequireAndVerifyClientCert
@@ -468,6 +478,7 @@ func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
 		return err
 	}
 
+	serverMu.Lock()
 	// create secure HTTP server
 	remoteAdminServer = &http.Server{
 		Addr:              addr.String(), // for logging purposes only
@@ -479,6 +490,7 @@ func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
 		MaxHeaderBytes:    1024 * 64,
 		ErrorLog:          serverLogger,
 	}
+	serverMu.Unlock()
 
 	// start listener
 	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
@@ -488,7 +500,10 @@ func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
 	ln = tls.NewListener(ln, tlsConfig)
 
 	go func() {
-		if err := remoteAdminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+		serverMu.Lock()
+		server := remoteAdminServer
+		serverMu.Unlock()
+		if err := server.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
 			remoteLogger.Error("admin remote server shutdown for unknown reason", zap.Error(err))
 		}
 	}()
@@ -499,7 +514,7 @@ func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
 	return nil
 }
 
-func (ident *IdentityConfig) certmagicConfig(logger *zap.Logger) *certmagic.Config {
+func (ident *IdentityConfig) certmagicConfig(logger *zap.Logger, makeCache bool) *certmagic.Config {
 	if ident == nil {
 		// user might not have configured identity; that's OK, we can still make a
 		// certmagic config, although it'll be mostly useless for remote management
@@ -510,7 +525,7 @@ func (ident *IdentityConfig) certmagicConfig(logger *zap.Logger) *certmagic.Conf
 		Logger:  logger,
 		Issuers: ident.issuers,
 	}
-	if identityCertCache == nil {
+	if makeCache {
 		identityCertCache = certmagic.NewCache(certmagic.CacheOptions{
 			GetConfigForCert: func(certmagic.Certificate) (*certmagic.Config, error) {
 				return cmCfg, nil
@@ -533,7 +548,7 @@ func (ctx Context) IdentityCredentials(logger *zap.Logger) ([]tls.Certificate, e
 	if logger == nil {
 		logger = Log()
 	}
-	magic := ident.certmagicConfig(logger)
+	magic := ident.certmagicConfig(logger, false)
 	return magic.ClientCredentials(ctx, ident.Identifiers)
 }
 
@@ -1186,7 +1201,7 @@ var (
 // will get deleted before the process gracefully exits.
 func PIDFile(filename string) error {
 	pid := []byte(strconv.Itoa(os.Getpid()) + "\n")
-	err := ioutil.WriteFile(filename, pid, 0600)
+	err := os.WriteFile(filename, pid, 0600)
 	if err != nil {
 		return err
 	}
@@ -1223,6 +1238,7 @@ var bufPool = sync.Pool{
 
 // keep a reference to admin endpoint singletons while they're active
 var (
+	serverMu                            sync.Mutex
 	localAdminServer, remoteAdminServer *http.Server
 	identityCertCache                   *certmagic.Cache
 )

admin_test.go

@@ -17,9 +17,28 @@ package caddy
 import (
 	"encoding/json"
 	"reflect"
+	"sync"
 	"testing"
 )
 
+var testCfg = []byte(`{
+			"apps": {
+				"http": {
+					"servers": {
+						"myserver": {
+							"listen": ["tcp/localhost:8080-8084"],
+							"read_timeout": "30s"
+						},
+						"yourserver": {
+							"listen": ["127.0.0.1:5000"],
+							"read_header_timeout": "15s"
+						}
+					}
+				}
+			}
+		}
+		`)
+
 func TestUnsyncedConfigAccess(t *testing.T) {
 	// each test is performed in sequence, so
 	// each change builds on the previous ones;
@@ -108,25 +127,24 @@ func TestUnsyncedConfigAccess(t *testing.T) {
 	}
 }
 
+// TestLoadConcurrent exercises Load under concurrent conditions
+// and is most useful under test with `-race` enabled.
+func TestLoadConcurrent(t *testing.T) {
+	var wg sync.WaitGroup
+
+	for i := 0; i < 100; i++ {
+		wg.Add(1)
+		go func() {
+			_ = Load(testCfg, true)
+			wg.Done()
+		}()
+	}
+
+	wg.Wait()
+}
+
 func BenchmarkLoad(b *testing.B) {
 	for i := 0; i < b.N; i++ {
-		cfg := []byte(`{
-			"apps": {
-				"http": {
-					"servers": {
-						"myserver": {
-							"listen": ["tcp/localhost:8080-8084"],
-							"read_timeout": "30s"
-						},
-						"yourserver": {
-							"listen": ["127.0.0.1:5000"],
-							"read_header_timeout": "15s"
-						}
-					}
-				}
-			}
-		}
-		`)
-		Load(cfg, true)
+		Load(testCfg, true)
 	}
 }

caddy.go

@@ -20,7 +20,6 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
 	"net/http"
 	"os"
@@ -268,8 +267,9 @@ func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 		newCfg != nil &&
 		newCfg.Admin != nil &&
 		newCfg.Admin.Config != nil &&
-		newCfg.Admin.Config.LoadRaw != nil {
-		return fmt.Errorf("recursive config loading detected: pulled configs cannot pull other configs")
+		newCfg.Admin.Config.LoadRaw != nil &&
+		newCfg.Admin.Config.LoadInterval <= 0 {
+		return fmt.Errorf("recursive config loading detected: pulled configs cannot pull other configs without positive load_interval")
 	}
 
 	// run the new config and start all its apps
@@ -299,7 +299,7 @@ func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 				zap.String("dir", dir),
 				zap.Error(err))
 		} else {
-			err := ioutil.WriteFile(ConfigAutosavePath, cfgJSON, 0600)
+			err := os.WriteFile(ConfigAutosavePath, cfgJSON, 0600)
 			if err == nil {
 				Log().Info("autosaved config (load with --resume flag)", zap.String("file", ConfigAutosavePath))
 			} else {
@@ -480,23 +480,42 @@ func finishSettingUp(ctx Context, cfg *Config) error {
 		if err != nil {
 			return fmt.Errorf("loading config loader module: %s", err)
 		}
-		loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
-		if err != nil {
-			return fmt.Errorf("loading dynamic config from %T: %v", val, err)
-		}
-
-		// do this in a goroutine so current config can finish being loaded; otherwise deadlock
-		go func() {
-			Log().Info("applying dynamically-loaded config", zap.String("loader_module", val.(Module).CaddyModule().ID.Name()))
+		runLoadedConfig := func(config []byte) {
+			Log().Info("applying dynamically-loaded config", zap.String("loader_module", val.(Module).CaddyModule().ID.Name()), zap.Int("pull_interval", int(cfg.Admin.Config.LoadInterval)))
 			currentCfgMu.Lock()
-			err := unsyncedDecodeAndRun(loadedConfig, false)
+			err := unsyncedDecodeAndRun(config, false)
 			currentCfgMu.Unlock()
 			if err == nil {
 				Log().Info("dynamically-loaded config applied successfully")
 			} else {
 				Log().Error("running dynamically-loaded config failed", zap.Error(err))
 			}
-		}()
+		}
+		if cfg.Admin.Config.LoadInterval > 0 {
+			go func() {
+				select {
+				// if LoadInterval is positive, will wait for the interval and then run with new config
+				case <-time.After(time.Duration(cfg.Admin.Config.LoadInterval)):
+					loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+					if err != nil {
+						Log().Error("loading dynamic config failed", zap.Error(err))
+						return
+					}
+					runLoadedConfig(loadedConfig)
+				case <-ctx.Done():
+					return
+				}
+			}()
+		} else {
+			// if no LoadInterval is provided, will load config synchronously
+			loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+			if err != nil {
+				return fmt.Errorf("loading dynamic config from %T: %v", val, err)
+			}
+			// do this in a goroutine so current config can finish being loaded; otherwise deadlock
+			go runLoadedConfig(loadedConfig)
+		}
+
 	}
 
 	return nil
@@ -680,13 +699,13 @@ func ParseDuration(s string) (time.Duration, error) {
 // have its own unique ID.
 func InstanceID() (uuid.UUID, error) {
 	uuidFilePath := filepath.Join(AppDataDir(), "instance.uuid")
-	uuidFileBytes, err := ioutil.ReadFile(uuidFilePath)
+	uuidFileBytes, err := os.ReadFile(uuidFilePath)
 	if os.IsNotExist(err) {
 		uuid, err := uuid.NewRandom()
 		if err != nil {
 			return uuid, err
 		}
-		err = ioutil.WriteFile(uuidFilePath, []byte(uuid.String()), 0600)
+		err = os.WriteFile(uuidFilePath, []byte(uuid.String()), 0600)
 		return uuid, err
 	} else if err != nil {
 		return [16]byte{}, err

caddyconfig/caddyfile/dispenser.go

@@ -345,13 +345,13 @@ func (d *Dispenser) EOFErr() error {
 
 // Err generates a custom parse-time error with a message of msg.
 func (d *Dispenser) Err(msg string) error {
-	msg = fmt.Sprintf("%s:%d - Error during parsing: %s", d.File(), d.Line(), msg)
-	return errors.New(msg)
+	return d.Errf(msg)
 }
 
 // Errf is like Err, but for formatted error messages
 func (d *Dispenser) Errf(format string, args ...interface{}) error {
-	return d.Err(fmt.Sprintf(format, args...))
+	err := fmt.Errorf(format, args...)
+	return fmt.Errorf("%s:%d - Error during parsing: %w", d.File(), d.Line(), err)
 }
 
 // Delete deletes the current token and returns the updated slice

caddyconfig/caddyfile/dispenser_test.go

@@ -15,6 +15,7 @@
 package caddyfile
 
 import (
+	"errors"
 	"reflect"
 	"strings"
 	"testing"
@@ -303,4 +304,10 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 	if !strings.Contains(err.Error(), "foobar") {
 		t.Errorf("Expected error message with custom message in it ('foobar'); got '%v'", err)
 	}
+
+	var ErrBarIsFull = errors.New("bar is full")
+	bookingError := d.Errf("unable to reserve: %w", ErrBarIsFull)
+	if !errors.Is(bookingError, ErrBarIsFull) {
+		t.Errorf("Errf(): should be able to unwrap the error chain")
+	}
 }

caddyconfig/caddyfile/lexer.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

caddyconfig/caddyfile/lexer_test.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

caddyconfig/caddyfile/parse.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -17,7 +17,7 @@ package caddyfile
 import (
 	"bytes"
 	"fmt"
-	"io/ioutil"
+	"io"
 	"log"
 	"os"
 	"path/filepath"
@@ -211,6 +211,12 @@ func (p *parser) addresses() error {
 			if expectingAnother {
 				return p.Errf("Expected another address but had '%s' - check for extra comma", tkn)
 			}
+			// Mark this server block as being defined with braces.
+			// This is used to provide a better error message when
+			// the user may have tried to define two server blocks
+			// without having used braces, which are required in
+			// that case.
+			p.block.HasBraces = true
 			break
 		}
 
@@ -229,6 +235,13 @@ func (p *parser) addresses() error {
 				expectingAnother = false // but we may still see another one on this line
 			}
 
+			// If there's a comma here, it's probably because they didn't use a space
+			// between their two domains, e.g. "foo.com,bar.com", which would not be
+			// parsed as two separate site addresses.
+			if strings.Contains(tkn, ",") {
+				return p.Errf("Site addresses cannot contain a comma ',': '%s' - put a space after the comma to separate site addresses", tkn)
+			}
+
 			p.block.Keys = append(p.block.Keys, tkn)
 		}
 
@@ -434,7 +447,7 @@ func (p *parser) doSingleImport(importFile string) ([]Token, error) {
 		return nil, p.Errf("Could not import %s: is a directory", importFile)
 	}
 
-	input, err := ioutil.ReadAll(file)
+	input, err := io.ReadAll(file)
 	if err != nil {
 		return nil, p.Errf("Could not read imported file %s: %v", importFile, err)
 	}
@@ -564,8 +577,9 @@ func (p *parser) snippetTokens() ([]Token, error) {
 // head of the server block with tokens, which are
 // grouped by segments.
 type ServerBlock struct {
-	Keys     []string
-	Segments []Segment
+	HasBraces bool
+	Keys      []string
+	Segments  []Segment
 }
 
 // DispenseDirective returns a dispenser that contains

caddyconfig/caddyfile/parse_test.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -16,7 +16,6 @@ package caddyfile
 
 import (
 	"bytes"
-	"io/ioutil"
 	"os"
 	"path/filepath"
 	"testing"
@@ -280,7 +279,7 @@ func TestRecursiveImport(t *testing.T) {
 	}
 
 	// test relative recursive import
-	err = ioutil.WriteFile(recursiveFile1, []byte(
+	err = os.WriteFile(recursiveFile1, []byte(
 		`localhost
 		dir1
 		import recursive_import_test2`), 0644)
@@ -289,7 +288,7 @@ func TestRecursiveImport(t *testing.T) {
 	}
 	defer os.Remove(recursiveFile1)
 
-	err = ioutil.WriteFile(recursiveFile2, []byte("dir2 1"), 0644)
+	err = os.WriteFile(recursiveFile2, []byte("dir2 1"), 0644)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -314,7 +313,7 @@ func TestRecursiveImport(t *testing.T) {
 	}
 
 	// test absolute recursive import
-	err = ioutil.WriteFile(recursiveFile1, []byte(
+	err = os.WriteFile(recursiveFile1, []byte(
 		`localhost
 		dir1
 		import `+recursiveFile2), 0644)
@@ -370,7 +369,7 @@ func TestDirectiveImport(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	err = ioutil.WriteFile(directiveFile, []byte(`prop1 1
+	err = os.WriteFile(directiveFile, []byte(`prop1 1
 	prop2 2`), 0644)
 	if err != nil {
 		t.Fatal(err)
@@ -633,7 +632,7 @@ func TestSnippets(t *testing.T) {
 }
 
 func writeStringToTempFileOrDie(t *testing.T, str string) (pathToFile string) {
-	file, err := ioutil.TempFile("", t.Name())
+	file, err := os.CreateTemp("", t.Name())
 	if err != nil {
 		panic(err) // get a stack trace so we know where this was called from.
 	}

caddyconfig/httpcaddyfile/addresses.go

@@ -337,7 +337,9 @@ func (a Address) Normalize() Address {
 	// ensure host is normalized if it's an IP address
 	host := strings.TrimSpace(a.Host)
 	if ip := net.ParseIP(host); ip != nil {
-		host = ip.String()
+		if ipv6 := ip.To16(); ipv6 != nil && ipv6.DefaultMask() == nil {
+			host = ipv6.String()
+		}
 	}
 
 	return Address{
@@ -349,28 +351,6 @@ func (a Address) Normalize() Address {
 	}
 }
 
-// Key returns a string form of a, much like String() does, but this
-// method doesn't add anything default that wasn't in the original.
-func (a Address) Key() string {
-	res := ""
-	if a.Scheme != "" {
-		res += a.Scheme + "://"
-	}
-	if a.Host != "" {
-		res += a.Host
-	}
-	// insert port only if the original has its own explicit port
-	if a.Port != "" &&
-		len(a.Original) >= len(res) &&
-		strings.HasPrefix(a.Original[len(res):], ":"+a.Port) {
-		res += ":" + a.Port
-	}
-	if a.Path != "" {
-		res += a.Path
-	}
-	return res
-}
-
 // lowerExceptPlaceholders lowercases s except within
 // placeholders (substrings in non-escaped '{ }' spans).
 // See https://github.com/caddyserver/caddy/issues/3264

caddyconfig/httpcaddyfile/addresses_test.go

@@ -106,67 +106,128 @@ func TestAddressString(t *testing.T) {
 func TestKeyNormalization(t *testing.T) {
 	testCases := []struct {
 		input  string
-		expect string
+		expect Address
 	}{
 		{
-			input:  "example.com",
-			expect: "example.com",
+			input: "example.com",
+			expect: Address{
+				Host: "example.com",
+			},
 		},
 		{
-			input:  "http://host:1234/path",
-			expect: "http://host:1234/path",
+			input: "http://host:1234/path",
+			expect: Address{
+				Scheme: "http",
+				Host:   "host",
+				Port:   "1234",
+				Path:   "/path",
+			},
 		},
 		{
-			input:  "HTTP://A/ABCDEF",
-			expect: "http://a/ABCDEF",
+			input: "HTTP://A/ABCDEF",
+			expect: Address{
+				Scheme: "http",
+				Host:   "a",
+				Path:   "/ABCDEF",
+			},
 		},
 		{
-			input:  "A/ABCDEF",
-			expect: "a/ABCDEF",
+			input: "A/ABCDEF",
+			expect: Address{
+				Host: "a",
+				Path: "/ABCDEF",
+			},
 		},
 		{
-			input:  "A:2015/Path",
-			expect: "a:2015/Path",
+			input: "A:2015/Path",
+			expect: Address{
+				Host: "a",
+				Port: "2015",
+				Path: "/Path",
+			},
 		},
 		{
-			input:  "sub.{env.MY_DOMAIN}",
-			expect: "sub.{env.MY_DOMAIN}",
+			input: "sub.{env.MY_DOMAIN}",
+			expect: Address{
+				Host: "sub.{env.MY_DOMAIN}",
+			},
 		},
 		{
-			input:  "sub.ExAmPle",
-			expect: "sub.example",
+			input: "sub.ExAmPle",
+			expect: Address{
+				Host: "sub.example",
+			},
 		},
 		{
-			input:  "sub.\\{env.MY_DOMAIN\\}",
-			expect: "sub.\\{env.my_domain\\}",
+			input: "sub.\\{env.MY_DOMAIN\\}",
+			expect: Address{
+				Host: "sub.\\{env.my_domain\\}",
+			},
 		},
 		{
-			input:  "sub.{env.MY_DOMAIN}.com",
-			expect: "sub.{env.MY_DOMAIN}.com",
+			input: "sub.{env.MY_DOMAIN}.com",
+			expect: Address{
+				Host: "sub.{env.MY_DOMAIN}.com",
+			},
 		},
 		{
-			input:  ":80",
-			expect: ":80",
+			input: ":80",
+			expect: Address{
+				Port: "80",
+			},
 		},
 		{
-			input:  ":443",
-			expect: ":443",
+			input: ":443",
+			expect: Address{
+				Port: "443",
+			},
 		},
 		{
-			input:  ":1234",
-			expect: ":1234",
+			input: ":1234",
+			expect: Address{
+				Port: "1234",
+			},
 		},
 		{
 			input:  "",
-			expect: "",
+			expect: Address{},
 		},
 		{
 			input:  ":",
-			expect: "",
+			expect: Address{},
 		},
 		{
-			input:  "[::]",
-			expect: "::",
+			input: "[::]",
+			expect: Address{
+				Host: "::",
+			},
+		},
+		{
+			input: "127.0.0.1",
+			expect: Address{
+				Host: "127.0.0.1",
+			},
+		},
+		{
+			input: "[2001:db8:85a3:8d3:1319:8a2e:370:7348]:1234",
+			expect: Address{
+				Host: "2001:db8:85a3:8d3:1319:8a2e:370:7348",
+				Port: "1234",
+			},
+		},
+		{
+			// IPv4 address in IPv6 form (#4381)
+			input: "[::ffff:cff4:e77d]:1234",
+			expect: Address{
+				Host: "::ffff:cff4:e77d",
+				Port: "1234",
+			},
+		},
+		{
+			input: "::ffff:cff4:e77d",
+			expect: Address{
+				Host: "::ffff:cff4:e77d",
+			},
 		},
 	}
 	for i, tc := range testCases {
@@ -175,9 +236,18 @@ func TestKeyNormalization(t *testing.T) {
 			t.Errorf("Test %d: Parsing address '%s': %v", i, tc.input, err)
 			continue
 		}
-		if actual := addr.Normalize().Key(); actual != tc.expect {
-			t.Errorf("Test %d: Input '%s': Expected '%s' but got '%s'", i, tc.input, tc.expect, actual)
+		actual := addr.Normalize()
+		if actual.Scheme != tc.expect.Scheme {
+			t.Errorf("Test %d: Input '%s': Expected Scheme='%s' but got Scheme='%s'", i, tc.input, tc.expect.Scheme, actual.Scheme)
+		}
+		if actual.Host != tc.expect.Host {
+			t.Errorf("Test %d: Input '%s': Expected Host='%s' but got Host='%s'", i, tc.input, tc.expect.Host, actual.Host)
+		}
+		if actual.Port != tc.expect.Port {
+			t.Errorf("Test %d: Input '%s': Expected Port='%s' but got Port='%s'", i, tc.input, tc.expect.Port, actual.Port)
+		}
+		if actual.Path != tc.expect.Path {
+			t.Errorf("Test %d: Input '%s': Expected Path='%s' but got Path='%s'", i, tc.input, tc.expect.Path, actual.Path)
 		}
-
 	}
 }

caddyconfig/httpcaddyfile/builtins.go

@@ -19,8 +19,8 @@ import (
 	"encoding/pem"
 	"fmt"
 	"html"
-	"io/ioutil"
 	"net/http"
+	"os"
 	"reflect"
 	"strconv"
 	"strings"
@@ -230,7 +230,7 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 							return nil, h.ArgErr()
 						}
 						filename := h.Val()
-						certDataPEM, err := ioutil.ReadFile(filename)
+						certDataPEM, err := os.ReadFile(filename)
 						if err != nil {
 							return nil, err
 						}

caddyconfig/httpcaddyfile/directives.go

@@ -44,9 +44,9 @@ var directiveOrder = []string{
 	"request_body",
 
 	"redir",
-	"rewrite",
 
 	// URI manipulation
+	"rewrite",
 	"uri",
 	"try_files",
 
@@ -54,23 +54,23 @@ var directiveOrder = []string{
 	"basicauth",
 	"request_header",
 	"encode",
+	"push",
 	"templates",
 
 	// special routing & dispatching directives
 	"handle",
 	"handle_path",
 	"route",
-	"push",
 
 	// handlers that typically respond to requests
+	"abort",
+	"error",
 	"respond",
 	"metrics",
 	"reverse_proxy",
 	"php_fastcgi",
 	"file_server",
 	"acme_server",
-	"abort",
-	"error",
 }
 
 // directiveIsOrdered returns true if dir is
@@ -329,7 +329,7 @@ func parseSegmentAsConfig(h Helper) ([]ConfigValue, error) {
 			dir := seg.Directive()
 			dirFunc, ok := registeredDirectives[dir]
 			if !ok {
-				return nil, h.Errf("unrecognized directive: %s", dir)
+				return nil, h.Errf("unrecognized directive: %s - are you sure your Caddyfile structure (nesting and braces) is correct?", dir)
 			}
 
 			subHelper := h
@@ -478,6 +478,27 @@ func (sb serverBlock) hostsFromKeys(loggerMode bool) []string {
 	return sblockHosts
 }
 
+func (sb serverBlock) hostsFromKeysNotHTTP(httpPort string) []string {
+	// ensure each entry in our list is unique
+	hostMap := make(map[string]struct{})
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			continue
+		}
+		if addr.Scheme != "http" || addr.Port != httpPort {
+			hostMap[addr.Host] = struct{}{}
+		}
+	}
+
+	// convert map to slice
+	sblockHosts := make([]string, 0, len(hostMap))
+	for host := range hostMap {
+		sblockHosts = append(sblockHosts, host)
+	}
+
+	return sblockHosts
+}
+
 // hasHostCatchAllKey returns true if sb has a key that
 // omits a host portion, i.e. it "catches all" hosts.
 func (sb serverBlock) hasHostCatchAllKey() bool {

caddyconfig/httpcaddyfile/httptype.go

@@ -113,6 +113,8 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 		"{tls_client_serial}", "{http.request.tls.client.serial}",
 		"{tls_client_subject}", "{http.request.tls.client.subject}",
 		"{tls_client_certificate_pem}", "{http.request.tls.client.certificate_pem}",
+		"{tls_client_certificate_der_base64}", "{http.request.tls.client.certificate_der_base64}",
+		"{upstream_hostport}", "{http.reverse_proxy.upstream.hostport}",
 	)
 
 	// these are placeholders that allow a user-defined final
@@ -169,7 +171,11 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 			dirFunc, ok := registeredDirectives[dir]
 			if !ok {
 				tkn := segment[0]
-				return nil, warnings, fmt.Errorf("%s:%d: unrecognized directive: %s", tkn.File, tkn.Line, dir)
+				message := "%s:%d: unrecognized directive: %s"
+				if !sb.block.HasBraces {
+					message += "\nDid you mean to define a second site? If so, you must use curly braces around each site to separate their configurations."
+				}
+				return nil, warnings, fmt.Errorf(message, tkn.File, tkn.Line, dir)
 			}
 
 			h := Helper{
@@ -522,6 +528,16 @@ func (st *ServerType) serversFromPairings(
 			}
 		}
 
+		// if needed, the ServerLogConfig is initialized beforehand so
+		// that all server blocks can populate it with data, even when not
+		// coming with a log directive
+		for _, sblock := range p.serverBlocks {
+			if len(sblock.pile["custom_log"]) != 0 {
+				srv.Logs = new(caddyhttp.ServerLogConfig)
+				break
+			}
+		}
+
 		// create a subroute for each site in the server block
 		for _, sblock := range p.serverBlocks {
 			matcherSetsEnc, err := st.compileEncodedMatcherSets(sblock)
@@ -636,9 +652,6 @@ func (st *ServerType) serversFromPairings(
 			sblockLogHosts := sblock.hostsFromKeys(true)
 			for _, cval := range sblock.pile["custom_log"] {
 				ncl := cval.Value.(namedCustomLog)
-				if srv.Logs == nil {
-					srv.Logs = new(caddyhttp.ServerLogConfig)
-				}
 				if sblock.hasHostCatchAllKey() {
 					// all requests for hosts not able to be listed should use
 					// this log because it's a catch-all-hosts server block

caddyconfig/httpcaddyfile/options.go

@@ -39,6 +39,7 @@ func init() {
 	RegisterGlobalOption("acme_dns", parseOptACMEDNS)
 	RegisterGlobalOption("acme_eab", parseOptACMEEAB)
 	RegisterGlobalOption("cert_issuer", parseOptCertIssuer)
+	RegisterGlobalOption("skip_install_trust", parseOptTrue)
 	RegisterGlobalOption("email", parseOptSingleString)
 	RegisterGlobalOption("admin", parseOptAdmin)
 	RegisterGlobalOption("on_demand_tls", parseOptOnDemand)
@@ -48,6 +49,7 @@ func init() {
 	RegisterGlobalOption("servers", parseServerOptions)
 	RegisterGlobalOption("ocsp_stapling", parseOCSPStaplingOptions)
 	RegisterGlobalOption("log", parseLogOptions)
+	RegisterGlobalOption("preferred_chains", parseOptPreferredChains)
 }
 
 func parseOptTrue(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) { return true, nil }
@@ -451,3 +453,8 @@ func parseLogOptions(d *caddyfile.Dispenser, existingVal interface{}) (interface
 
 	return configValues, nil
 }
+
+func parseOptPreferredChains(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	d.Next()
+	return caddytls.ParseCaddyfilePreferredChainsOptions(d)
+}

caddyconfig/httpcaddyfile/pkiapp.go

@@ -27,15 +27,35 @@ func (st ServerType) buildPKIApp(
 
 	pkiApp := &caddypki.PKI{CAs: make(map[string]*caddypki.CA)}
 
+	skipInstallTrust := false
+	if _, ok := options["skip_install_trust"]; ok {
+		skipInstallTrust = true
+	}
+	falseBool := false
+
 	for _, p := range pairings {
 		for _, sblock := range p.serverBlocks {
 			// find all the CAs that were defined and add them to the app config
+			// i.e. from any "acme_server" directives
 			for _, caCfgValue := range sblock.pile["pki.ca"] {
 				ca := caCfgValue.Value.(*caddypki.CA)
+				if skipInstallTrust {
+					ca.InstallTrust = &falseBool
+				}
 				pkiApp.CAs[ca.ID] = ca
 			}
 		}
 	}
 
+	// if there was no CAs defined in any of the servers,
+	// and we were requested to not install trust, then
+	// add one for the default/local CA to do so
+	if len(pkiApp.CAs) == 0 && skipInstallTrust {
+		ca := new(caddypki.CA)
+		ca.ID = caddypki.DefaultCAID
+		ca.InstallTrust = &falseBool
+		pkiApp.CAs[ca.ID] = ca
+	}
+
 	return pkiApp, warnings, nil
 }

caddyconfig/httpcaddyfile/tlsapp.go

@@ -189,7 +189,7 @@ func (st ServerType) buildTLSApp(
 			}
 
 			// associate our new automation policy with this server block's hosts
-			ap.Subjects = sblockHosts
+			ap.Subjects = sblock.hostsFromKeysNotHTTP(httpPort)
 			sort.Strings(ap.Subjects) // solely for deterministic test results
 
 			// if a combination of public and internal names were given
@@ -211,7 +211,7 @@ func (st ServerType) buildTLSApp(
 					// it that we would need to check here) since the hostname is known at handshake;
 					// and it is unexpected to switch to internal issuer when the user wants to get
 					// regular certificates on-demand for a class of certs like *.*.tld.
-					if !certmagic.SubjectIsIP(s) && !certmagic.SubjectIsInternal(s) && (strings.Count(s, "*.") < 2 || ap.OnDemand) {
+					if subjectQualifiesForPublicCert(ap, s) {
 						external = append(external, s)
 					} else {
 						internal = append(internal, s)
@@ -321,10 +321,15 @@ func (st ServerType) buildTLSApp(
 		globalACMECARoot := options["acme_ca_root"]
 		globalACMEDNS := options["acme_dns"]
 		globalACMEEAB := options["acme_eab"]
-		hasGlobalACMEDefaults := globalEmail != nil || globalACMECA != nil || globalACMECARoot != nil || globalACMEDNS != nil || globalACMEEAB != nil
+		globalPreferredChains := options["preferred_chains"]
+		hasGlobalACMEDefaults := globalEmail != nil || globalACMECA != nil || globalACMECARoot != nil || globalACMEDNS != nil || globalACMEEAB != nil || globalPreferredChains != nil
 		if hasGlobalACMEDefaults {
-			for _, ap := range tlsApp.Automation.Policies {
-				if len(ap.Issuers) == 0 {
+			// for _, ap := range tlsApp.Automation.Policies {
+			for i := 0; i < len(tlsApp.Automation.Policies); i++ {
+				ap := tlsApp.Automation.Policies[i]
+				if len(ap.Issuers) == 0 && automationPolicyHasAllPublicNames(ap) {
+					// for public names, create default issuers which will later be filled in with configured global defaults
+					// (internal names will implicitly use the internal issuer at auto-https time)
 					ap.Issuers = caddytls.DefaultIssuers()
 
 					// if a specific endpoint is configured, can't use multiple default issuers
@@ -405,6 +410,7 @@ func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]interf
 	globalACMECARoot := options["acme_ca_root"]
 	globalACMEDNS := options["acme_dns"]
 	globalACMEEAB := options["acme_eab"]
+	globalPreferredChains := options["preferred_chains"]
 
 	if globalEmail != nil && acmeIssuer.Email == "" {
 		acmeIssuer.Email = globalEmail.(string)
@@ -425,6 +431,9 @@ func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]interf
 	if globalACMEEAB != nil && acmeIssuer.ExternalAccount == nil {
 		acmeIssuer.ExternalAccount = globalACMEEAB.(*acme.EAB)
 	}
+	if globalPreferredChains != nil && acmeIssuer.PreferredChains == nil {
+		acmeIssuer.PreferredChains = globalPreferredChains.(*caddytls.ChainPreference)
+	}
 	return nil
 }
 
@@ -489,16 +498,23 @@ func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls
 	})
 
 	emptyAPCount := 0
+	origLenAPs := len(aps)
 	// compute the number of empty policies (disregarding subjects) - see #4128
 	emptyAP := new(caddytls.AutomationPolicy)
 	for i := 0; i < len(aps); i++ {
 		emptyAP.Subjects = aps[i].Subjects
 		if reflect.DeepEqual(aps[i], emptyAP) {
 			emptyAPCount++
+			if !automationPolicyHasAllPublicNames(aps[i]) {
+				// if this automation policy has internal names, we might as well remove it
+				// so auto-https can implicitly use the internal issuer
+				aps = append(aps[:i], aps[i+1:]...)
+				i--
+			}
 		}
 	}
 	// If all policies are empty, we can return nil, as there is no need to set any policy
-	if emptyAPCount == len(aps) {
+	if emptyAPCount == origLenAPs {
 		return nil
 	}
 
@@ -596,3 +612,21 @@ func automationPolicyShadows(i int, aps []*caddytls.AutomationPolicy) int {
 	}
 	return -1
 }
+
+// subjectQualifiesForPublicCert is like certmagic.SubjectQualifiesForPublicCert() except
+// that this allows domains with multiple wildcard levels like '*.*.example.com' to qualify
+// if the automation policy has OnDemand enabled (i.e. this function is more lenient).
+func subjectQualifiesForPublicCert(ap *caddytls.AutomationPolicy, subj string) bool {
+	return !certmagic.SubjectIsIP(subj) &&
+		!certmagic.SubjectIsInternal(subj) &&
+		(strings.Count(subj, "*.") < 2 || ap.OnDemand)
+}
+
+func automationPolicyHasAllPublicNames(ap *caddytls.AutomationPolicy) bool {
+	for _, subj := range ap.Subjects {
+		if !subjectQualifiesForPublicCert(ap, subj) {
+			return false
+		}
+	}
+	return true
+}

caddyconfig/httploader.go

@@ -1,11 +1,26 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 package caddyconfig
 
 import (
 	"crypto/tls"
 	"crypto/x509"
 	"fmt"
-	"io/ioutil"
+	"io"
 	"net/http"
+	"os"
 	"time"
 
 	"github.com/caddyserver/caddy/v2"
@@ -81,7 +96,7 @@ func (hl HTTPLoader) LoadConfig(ctx caddy.Context) ([]byte, error) {
 		return nil, fmt.Errorf("server responded with HTTP %d", resp.StatusCode)
 	}
 
-	body, err := ioutil.ReadAll(resp.Body)
+	body, err := io.ReadAll(resp.Body)
 	if err != nil {
 		return nil, err
 	}
@@ -130,7 +145,7 @@ func (hl HTTPLoader) makeClient(ctx caddy.Context) (*http.Client, error) {
 		if len(hl.TLS.RootCAPEMFiles) > 0 {
 			rootPool := x509.NewCertPool()
 			for _, pemFile := range hl.TLS.RootCAPEMFiles {
-				pemData, err := ioutil.ReadFile(pemFile)
+				pemData, err := os.ReadFile(pemFile)
 				if err != nil {
 					return nil, fmt.Errorf("failed reading ca cert: %v", err)
 				}

caddytest/caddytest.go

@@ -7,7 +7,7 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
-	"io/ioutil"
+	"io"
 	"log"
 	"net"
 	"net/http"
@@ -129,7 +129,7 @@ func (tc *Tester) initServer(rawConfig string, configType string) error {
 				return
 			}
 			defer res.Body.Close()
-			body, _ := ioutil.ReadAll(res.Body)
+			body, _ := io.ReadAll(res.Body)
 
 			var out bytes.Buffer
 			_ = json.Indent(&out, body, "", "  ")
@@ -162,7 +162,7 @@ func (tc *Tester) initServer(rawConfig string, configType string) error {
 	timeElapsed(start, "caddytest: config load time")
 
 	defer res.Body.Close()
-	body, err := ioutil.ReadAll(res.Body)
+	body, err := io.ReadAll(res.Body)
 	if err != nil {
 		tc.t.Errorf("unable to read response. %s", err)
 		return err
@@ -202,7 +202,7 @@ func (tc *Tester) ensureConfigRunning(rawConfig string, configType string) error
 			return nil
 		}
 		defer resp.Body.Close()
-		actualBytes, err := ioutil.ReadAll(resp.Body)
+		actualBytes, err := io.ReadAll(resp.Body)
 		if err != nil {
 			return nil
 		}
@@ -471,7 +471,7 @@ func (tc *Tester) AssertResponse(req *http.Request, expectedStatusCode int, expe
 	resp := tc.AssertResponseCode(req, expectedStatusCode)
 
 	defer resp.Body.Close()
-	bytes, err := ioutil.ReadAll(resp.Body)
+	bytes, err := io.ReadAll(resp.Body)
 	if err != nil {
 		tc.t.Fatalf("unable to read the response body %s", err)
 	}

caddytest/integration/autohttps_test.go

@@ -103,3 +103,23 @@ func TestAutoHTTPRedirectsInsertedBeforeUserDefinedCatchAll(t *testing.T) {
 	tester.AssertGetResponse("http://foo.localhost:9080/", 200, "Foo")
 	tester.AssertGetResponse("http://baz.localhost:9080/", 200, "Baz")
 }
+
+func TestAutoHTTPRedirectsInsertedBeforeUserDefinedCatchAllWithNoExplicitHTTPSite(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+		local_certs
+	}
+	http://:9080 {
+		respond "Foo"
+	}
+	bar.localhost {
+		respond "Bar"
+	}
+  `, "caddyfile")
+	tester.AssertRedirect("http://bar.localhost:9080/", "https://bar.localhost/", http.StatusPermanentRedirect)
+	tester.AssertGetResponse("http://foo.localhost:9080/", 200, "Foo")
+	tester.AssertGetResponse("http://baz.localhost:9080/", 200, "Foo")
+}

caddytest/integration/caddyfile_adapt/error_example.txt

@@ -0,0 +1,138 @@
+example.com {
+	root * /srv
+
+	# Trigger errors for certain paths
+	error /private* "Unauthorized" 403
+	error /hidden* "Not found" 404
+
+	# Handle the error by serving an HTML page 
+	handle_errors {
+		rewrite * /{http.error.status_code}.html
+		file_server
+	}
+
+	file_server
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "vars",
+													"root": "/srv"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"error": "Unauthorized",
+													"handler": "error",
+													"status_code": 403
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/private*"
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"error": "Not found",
+													"handler": "error",
+													"status_code": 404
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/hidden*"
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "file_server",
+													"hide": [
+														"./Caddyfile"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"errors": {
+						"routes": [
+							{
+								"match": [
+									{
+										"host": [
+											"example.com"
+										]
+									}
+								],
+								"handle": [
+									{
+										"handler": "subroute",
+										"routes": [
+											{
+												"group": "group0",
+												"handle": [
+													{
+														"handler": "rewrite",
+														"uri": "/{http.error.status_code}.html"
+													}
+												]
+											},
+											{
+												"handle": [
+													{
+														"handler": "file_server",
+														"hide": [
+															"./Caddyfile"
+														]
+													}
+												]
+											}
+										]
+									}
+								],
+								"terminal": true
+							}
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_disable_canonical_uris.txt

@@ -0,0 +1,32 @@
+:80
+
+file_server {
+	disable_canonical_uris
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"canonical_uris": false,
+									"handler": "file_server",
+									"hide": [
+										"./Caddyfile"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_preferred_chains.txt

@@ -0,0 +1,56 @@
+{
+	preferred_chains smallest
+}
+
+example.com
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"module": "acme",
+								"preferred_chains": {
+									"smallest": true
+								}
+							},
+							{
+								"module": "zerossl",
+								"preferred_chains": {
+									"smallest": true
+								}
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_skip_install_trust.txt

@@ -0,0 +1,56 @@
+{
+	skip_install_trust
+}
+
+a.example.com {
+	tls internal
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"pki": {
+			"certificate_authorities": {
+				"local": {
+					"install_trust": false
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"a.example.com"
+						],
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_skip_hosts.txt

@@ -0,0 +1,75 @@
+one.example.com {
+	log
+}
+
+two.example.com {
+}
+
+three.example.com {
+}
+
+example.com {
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"three.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"one.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"two.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"logs": {
+						"skip_hosts": [
+							"three.example.com",
+							"two.example.com",
+							"example.com"
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_handle_response.txt

@@ -0,0 +1,145 @@
+:8881 {
+	php_fastcgi app:9000 {
+		env FOO bar
+
+		@error status 4xx
+		handle_response @error {
+			root * /errors
+			rewrite * /{http.reverse_proxy.status_code}.html
+			file_server
+		}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8881"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}/index.php"
+										]
+									},
+									"not": [
+										{
+											"path": [
+												"*/"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"{http.request.uri.path}/"
+										]
+									},
+									"status_code": 308
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}",
+											"{http.request.uri.path}/index.php",
+											"index.php"
+										],
+										"split_path": [
+											".php"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"handler": "rewrite",
+									"uri": "{http.matchers.file.relative}"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"path": [
+										"*.php"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handle_response": [
+										{
+											"match": {
+												"status_code": [
+													4
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"handler": "vars",
+															"root": "/errors"
+														}
+													]
+												},
+												{
+													"group": "group0",
+													"handle": [
+														{
+															"handler": "rewrite",
+															"uri": "/{http.reverse_proxy.status_code}.html"
+														}
+													]
+												},
+												{
+													"handle": [
+														{
+															"handler": "file_server",
+															"hide": [
+																"./Caddyfile"
+															]
+														}
+													]
+												}
+											]
+										}
+									],
+									"handler": "reverse_proxy",
+									"transport": {
+										"env": {
+											"FOO": "bar"
+										},
+										"protocol": "fastcgi",
+										"split_path": [
+											".php"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "app:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_try_files_override.txt

@@ -0,0 +1,124 @@
+:8884
+
+php_fastcgi localhost:9000 {
+	# some php_fastcgi-specific subdirectives
+	split .php .php5
+	env VAR1 value1
+	env VAR2 value2
+	root /var/www
+	try_files {path} {path}/index.php =404
+	dial_timeout 3s
+	read_timeout 10s
+	write_timeout 20s
+
+	# passed through to reverse_proxy (directive order doesn't matter!)
+	lb_policy random
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}/index.php"
+										]
+									},
+									"not": [
+										{
+											"path": [
+												"*/"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"{http.request.uri.path}/"
+										]
+									},
+									"status_code": 308
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}",
+											"{http.request.uri.path}/index.php",
+											"=404"
+										],
+										"split_path": [
+											".php",
+											".php5"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"handler": "rewrite",
+									"uri": "{http.matchers.file.relative}"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"path": [
+										"*.php",
+										"*.php5"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"selection_policy": {
+											"policy": "random"
+										}
+									},
+									"transport": {
+										"dial_timeout": 3000000000,
+										"env": {
+											"VAR1": "value1",
+											"VAR2": "value2"
+										},
+										"protocol": "fastcgi",
+										"read_timeout": 10000000000,
+										"root": "/var/www",
+										"split_path": [
+											".php",
+											".php5"
+										],
+										"write_timeout": 20000000000
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_options.txt

@@ -21,7 +21,7 @@ https://example.com {
 			versions h2c 2
 			compression off
 			max_conns_per_host 5
-			max_idle_conns_per_host 2
+			keepalive_idle_conns_per_host 2
 		}
 	}
 }
@@ -79,8 +79,10 @@ https://example.com {
 														"dial_fallback_delay": 5000000000,
 														"dial_timeout": 3000000000,
 														"expect_continue_timeout": 9000000000,
+														"keep_alive": {
+															"max_idle_conns_per_host": 2
+														},
 														"max_conns_per_host": 5,
-														"max_idle_conns_per_host": 2,
 														"max_response_header_size": 30000000,
 														"protocol": "http",
 														"read_buffer_size": 10000000,

caddytest/integration/caddyfile_adapt/tls_acme_preferred_chains.txt

@@ -0,0 +1,57 @@
+localhost
+
+tls {
+	issuer acme {
+		preferred_chains {
+			any_common_name "Generic CA 1" "Generic CA 2"
+		}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"localhost"
+						],
+						"issuers": [
+							{
+								"module": "acme",
+								"preferred_chains": {
+									"any_common_name": [
+										"Generic CA 1",
+										"Generic CA 2"
+									]
+								}
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_7.txt

@@ -0,0 +1,68 @@
+# (this Caddyfile is contrived, but based on issues #4176 and #4198)
+
+http://example.com {
+}
+
+https://example.com {
+	tls internal
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_8.txt

@@ -0,0 +1,98 @@
+# (this Caddyfile is contrived, but based on issues #4176 and #4198)
+
+http://example.com {
+}
+
+https://example.com {
+	tls abc@example.com
+}
+
+http://localhost:8081 {
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv2": {
+					"listen": [
+						":8081"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"localhost"
+						]
+					}
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"email": "abc@example.com",
+								"module": "acme"
+							},
+							{
+								"email": "abc@example.com",
+								"module": "zerossl"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_global_email_localhost.txt

@@ -0,0 +1,66 @@
+{
+	email foo@bar
+}
+
+localhost {
+}
+
+example.com {
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"email": "foo@bar",
+								"module": "acme"
+							},
+							{
+								"email": "foo@bar",
+								"module": "zerossl"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_propagation_timeout.txt

@@ -0,0 +1,70 @@
+localhost
+
+respond "hello from localhost"
+tls {
+	issuer acme {
+		propagation_timeout "10m0s"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"localhost"
+						],
+						"issuers": [
+							{
+								"challenges": {
+									"dns": {
+										"propagation_timeout": 600000000000
+									}
+								},
+								"module": "acme"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt_test.go

@@ -3,7 +3,7 @@ package integration
 import (
 	jsonMod "encoding/json"
 	"fmt"
-	"io/ioutil"
+	"os"
 	"path/filepath"
 	"regexp"
 	"strings"
@@ -14,7 +14,7 @@ import (
 
 func TestCaddyfileAdaptToJSON(t *testing.T) {
 	// load the list of test files from the dir
-	files, err := ioutil.ReadDir("./caddyfile_adapt")
+	files, err := os.ReadDir("./caddyfile_adapt")
 	if err != nil {
 		t.Errorf("failed to read caddyfile_adapt dir: %s", err)
 	}
@@ -29,7 +29,7 @@ func TestCaddyfileAdaptToJSON(t *testing.T) {
 
 		// read the test file
 		filename := f.Name()
-		data, err := ioutil.ReadFile("./caddyfile_adapt/" + filename)
+		data, err := os.ReadFile("./caddyfile_adapt/" + filename)
 		if err != nil {
 			t.Errorf("failed to read %s dir: %s", filename, err)
 		}

caddytest/integration/reverseproxy_test.go

@@ -2,7 +2,6 @@ package integration
 
 import (
 	"fmt"
-	"io/ioutil"
 	"net"
 	"net/http"
 	"os"
@@ -85,7 +84,7 @@ func TestDialWithPlaceholderUnix(t *testing.T) {
 		t.SkipNow()
 	}
 
-	f, err := ioutil.TempFile("", "*.sock")
+	f, err := os.CreateTemp("", "*.sock")
 	if err != nil {
 		t.Errorf("failed to create TempFile: %s", err)
 		return
@@ -371,7 +370,7 @@ func TestReverseProxyHealthCheck(t *testing.T) {
 		reverse_proxy {
 			to localhost:2020
 	
-			health_path /health
+			health_uri /health
 			health_port 2021
 			health_interval 2s
 			health_timeout 5s
@@ -387,7 +386,7 @@ func TestReverseProxyHealthCheckUnixSocket(t *testing.T) {
 		t.SkipNow()
 	}
 	tester := caddytest.NewTester(t)
-	f, err := ioutil.TempFile("", "*.sock")
+	f, err := os.CreateTemp("", "*.sock")
 	if err != nil {
 		t.Errorf("failed to create TempFile: %s", err)
 		return
@@ -426,7 +425,7 @@ func TestReverseProxyHealthCheckUnixSocket(t *testing.T) {
 		reverse_proxy {
 			to unix/%s
 	
-			health_path /health
+			health_uri /health
 			health_port 2021
 			health_interval 2s
 			health_timeout 5s
@@ -436,3 +435,57 @@ func TestReverseProxyHealthCheckUnixSocket(t *testing.T) {
 
 	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
 }
+
+func TestReverseProxyHealthCheckUnixSocketWithoutPort(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.SkipNow()
+	}
+	tester := caddytest.NewTester(t)
+	f, err := os.CreateTemp("", "*.sock")
+	if err != nil {
+		t.Errorf("failed to create TempFile: %s", err)
+		return
+	}
+	// a hack to get a file name within a valid path to use as socket
+	socketName := f.Name()
+	os.Remove(f.Name())
+
+	server := http.Server{
+		Handler: http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+			if strings.HasPrefix(req.URL.Path, "/health") {
+				w.Write([]byte("ok"))
+				return
+			}
+			w.Write([]byte("Hello, World!"))
+		}),
+	}
+
+	unixListener, err := net.Listen("unix", socketName)
+	if err != nil {
+		t.Errorf("failed to listen on the socket: %s", err)
+		return
+	}
+	go server.Serve(unixListener)
+	t.Cleanup(func() {
+		server.Close()
+	})
+	runtime.Gosched() // Allow other goroutines to run
+
+	tester.InitServer(fmt.Sprintf(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:9080 {
+		reverse_proxy {
+			to unix/%s
+	
+			health_uri /health
+			health_interval 2s
+			health_timeout 5s
+		}
+	}
+	`, socketName), "caddyfile")
+
+	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
+}

caddytest/integration/stream_test.go

@@ -6,7 +6,6 @@ import (
 	"crypto/rand"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"net/http"
 	"net/http/httputil"
 	"net/url"
@@ -110,7 +109,7 @@ func TestH2ToH2CStream(t *testing.T) {
 	r, w := io.Pipe()
 	req := &http.Request{
 		Method: "PUT",
-		Body:   ioutil.NopCloser(r),
+		Body:   io.NopCloser(r),
 		URL: &url.URL{
 			Scheme: "https",
 			Host:   "127.0.0.1:9443",
@@ -134,7 +133,7 @@ func TestH2ToH2CStream(t *testing.T) {
 	}()
 
 	defer resp.Body.Close()
-	bytes, err := ioutil.ReadAll(resp.Body)
+	bytes, err := io.ReadAll(resp.Body)
 	if err != nil {
 		t.Fatalf("unable to read the response body %s", err)
 	}
@@ -319,7 +318,7 @@ func TestH2ToH1ChunkedResponse(t *testing.T) {
 	r, w := io.Pipe()
 	req := &http.Request{
 		Method: "PUT",
-		Body:   ioutil.NopCloser(r),
+		Body:   io.NopCloser(r),
 		URL: &url.URL{
 			Scheme: "https",
 			Host:   "127.0.0.1:9443",
@@ -342,7 +341,7 @@ func TestH2ToH1ChunkedResponse(t *testing.T) {
 	}
 
 	defer resp.Body.Close()
-	bytes, err := ioutil.ReadAll(resp.Body)
+	bytes, err := io.ReadAll(resp.Body)
 	if err != nil {
 		t.Fatalf("unable to read the response body %s", err)
 	}
@@ -370,7 +369,7 @@ func testH2ToH1ChunkedResponseServeH1(t *testing.T) *http.Server {
 		}
 
 		defer r.Body.Close()
-		bytes, err := ioutil.ReadAll(r.Body)
+		bytes, err := io.ReadAll(r.Body)
 		if err != nil {
 			t.Fatalf("unable to read the response body %s", err)
 		}

cmd/commandfuncs.go

@@ -19,16 +19,14 @@ import (
 	"context"
 	"crypto/rand"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
 	"net"
 	"net/http"
-	"net/url"
 	"os"
 	"os/exec"
-	"reflect"
 	"runtime"
 	"runtime/debug"
 	"sort"
@@ -121,7 +119,7 @@ func cmdStart(fl Flags) (int, error) {
 		for {
 			conn, err := ln.Accept()
 			if err != nil {
-				if !strings.Contains(err.Error(), "use of closed network connection") {
+				if !errors.Is(err, net.ErrClosed) {
 					log.Println(err)
 				}
 				break
@@ -182,7 +180,7 @@ func cmdRun(fl Flags) (int, error) {
 	var config []byte
 	var err error
 	if runCmdResumeFlag {
-		config, err = ioutil.ReadFile(caddy.ConfigAutosavePath)
+		config, err = os.ReadFile(caddy.ConfigAutosavePath)
 		if os.IsNotExist(err) {
 			// not a bad error; just can't resume if autosave file doesn't exist
 			caddy.Log().Info("no autosave file exists", zap.String("autosave_file", caddy.ConfigAutosavePath))
@@ -220,7 +218,7 @@ func cmdRun(fl Flags) (int, error) {
 	// if we are to report to another process the successful start
 	// of the server, do so now by echoing back contents of stdin
 	if runCmdPingbackFlag != "" {
-		confirmationBytes, err := ioutil.ReadAll(os.Stdin)
+		confirmationBytes, err := io.ReadAll(os.Stdin)
 		if err != nil {
 			return caddy.ExitCodeFailedStartup,
 				fmt.Errorf("reading confirmation bytes from stdin: %v", err)
@@ -332,7 +330,7 @@ func cmdReload(fl Flags) (int, error) {
 }
 
 func cmdVersion(_ Flags) (int, error) {
-	fmt.Println(caddyVersion())
+	fmt.Println(CaddyVersion())
 	return caddy.ExitCodeSuccess, nil
 }
 
@@ -362,6 +360,7 @@ func cmdBuildInfo(fl Flags) (int, error) {
 func cmdListModules(fl Flags) (int, error) {
 	packages := fl.Bool("packages")
 	versions := fl.Bool("versions")
+	skipStandard := fl.Bool("skip-standard")
 
 	printModuleInfo := func(mi moduleInfo) {
 		fmt.Print(mi.caddyModuleID)
@@ -390,14 +389,19 @@ func cmdListModules(fl Flags) (int, error) {
 		return caddy.ExitCodeSuccess, nil
 	}
 
-	if len(standard) > 0 {
-		for _, mod := range standard {
-			printModuleInfo(mod)
-		}
-	}
-	fmt.Printf("\n  Standard modules: %d\n", len(standard))
-	if len(nonstandard) > 0 {
+	// Standard modules (always shipped with Caddy)
+	if !skipStandard {
 		if len(standard) > 0 {
+			for _, mod := range standard {
+				printModuleInfo(mod)
+			}
+		}
+		fmt.Printf("\n  Standard modules: %d\n", len(standard))
+	}
+
+	// Non-standard modules (third party plugins)
+	if len(nonstandard) > 0 {
+		if len(standard) > 0 && !skipStandard {
 			fmt.Println()
 		}
 		for _, mod := range nonstandard {
@@ -405,8 +409,10 @@ func cmdListModules(fl Flags) (int, error) {
 		}
 	}
 	fmt.Printf("\n  Non-standard modules: %d\n", len(nonstandard))
+
+	// Unknown modules (couldn't get Caddy module info)
 	if len(unknown) > 0 {
-		if len(standard) > 0 || len(nonstandard) > 0 {
+		if (len(standard) > 0 && !skipStandard) || len(nonstandard) > 0 {
 			fmt.Println()
 		}
 		for _, mod := range unknown {
@@ -458,7 +464,7 @@ func cmdAdaptConfig(fl Flags) (int, error) {
 			fmt.Errorf("unrecognized config adapter: %s", adaptCmdAdapterFlag)
 	}
 
-	input, err := ioutil.ReadFile(adaptCmdInputFlag)
+	input, err := os.ReadFile(adaptCmdInputFlag)
 	if err != nil {
 		return caddy.ExitCodeFailedStartup,
 			fmt.Errorf("reading input file: %v", err)
@@ -542,7 +548,7 @@ func cmdFmt(fl Flags) (int, error) {
 
 	// as a special case, read from stdin if the file name is "-"
 	if formatCmdConfigFile == "-" {
-		input, err := ioutil.ReadAll(os.Stdin)
+		input, err := io.ReadAll(os.Stdin)
 		if err != nil {
 			return caddy.ExitCodeFailedStartup,
 				fmt.Errorf("reading stdin: %v", err)
@@ -551,7 +557,7 @@ func cmdFmt(fl Flags) (int, error) {
 		return caddy.ExitCodeSuccess, nil
 	}
 
-	input, err := ioutil.ReadFile(formatCmdConfigFile)
+	input, err := os.ReadFile(formatCmdConfigFile)
 	if err != nil {
 		return caddy.ExitCodeFailedStartup,
 			fmt.Errorf("reading input file: %v", err)
@@ -560,7 +566,7 @@ func cmdFmt(fl Flags) (int, error) {
 	output := caddyfile.Format(input)
 
 	if fl.Bool("overwrite") {
-		if err := ioutil.WriteFile(formatCmdConfigFile, output, 0600); err != nil {
+		if err := os.WriteFile(formatCmdConfigFile, output, 0600); err != nil {
 			return caddy.ExitCodeFailedStartup, nil
 		}
 	} else {
@@ -570,151 +576,6 @@ func cmdFmt(fl Flags) (int, error) {
 	return caddy.ExitCodeSuccess, nil
 }
 
-func cmdUpgrade(_ Flags) (int, error) {
-	l := caddy.Log()
-
-	thisExecPath, err := os.Executable()
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("determining current executable path: %v", err)
-	}
-	thisExecStat, err := os.Stat(thisExecPath)
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("retrieving current executable permission bits: %v", err)
-	}
-	l.Info("this executable will be replaced", zap.String("path", thisExecPath))
-
-	// get the list of nonstandard plugins
-	_, nonstandard, _, err := getModules()
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("unable to enumerate installed plugins: %v", err)
-	}
-	pluginPkgs := make(map[string]struct{})
-	for _, mod := range nonstandard {
-		if mod.goModule.Replace != nil {
-			return caddy.ExitCodeFailedStartup, fmt.Errorf("cannot auto-upgrade when Go module has been replaced: %s => %s",
-				mod.goModule.Path, mod.goModule.Replace.Path)
-		}
-		l.Info("found non-standard module",
-			zap.String("id", mod.caddyModuleID),
-			zap.String("package", mod.goModule.Path))
-		pluginPkgs[mod.goModule.Path] = struct{}{}
-	}
-
-	// build the request URL to download this custom build
-	qs := url.Values{
-		"os":   {runtime.GOOS},
-		"arch": {runtime.GOARCH},
-	}
-	for pkg := range pluginPkgs {
-		qs.Add("p", pkg)
-	}
-	urlStr := fmt.Sprintf("https://caddyserver.com/api/download?%s", qs.Encode())
-
-	// initiate the build
-	l.Info("requesting build",
-		zap.String("os", qs.Get("os")),
-		zap.String("arch", qs.Get("arch")),
-		zap.Strings("packages", qs["p"]))
-	resp, err := http.Get(urlStr)
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("secure request failed: %v", err)
-	}
-	defer resp.Body.Close()
-	if resp.StatusCode >= 400 {
-		var details struct {
-			StatusCode int `json:"status_code"`
-			Error      struct {
-				Message string `json:"message"`
-				ID      string `json:"id"`
-			} `json:"error"`
-		}
-		err2 := json.NewDecoder(resp.Body).Decode(&details)
-		if err2 != nil {
-			return caddy.ExitCodeFailedStartup, fmt.Errorf("download and error decoding failed: HTTP %d: %v", resp.StatusCode, err2)
-		}
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("download failed: HTTP %d: %s (id=%s)", resp.StatusCode, details.Error.Message, details.Error.ID)
-	}
-
-	// back up the current binary, in case something goes wrong we can replace it
-	backupExecPath := thisExecPath + ".tmp"
-	l.Info("build acquired; backing up current executable",
-		zap.String("current_path", thisExecPath),
-		zap.String("backup_path", backupExecPath))
-	err = os.Rename(thisExecPath, backupExecPath)
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("backing up current binary: %v", err)
-	}
-	defer func() {
-		if err != nil {
-			err2 := os.Rename(backupExecPath, thisExecPath)
-			if err2 != nil {
-				l.Error("restoring original executable failed; will need to be restored manually",
-					zap.String("backup_path", backupExecPath),
-					zap.String("original_path", thisExecPath),
-					zap.Error(err2))
-			}
-		}
-	}()
-
-	// download the file; do this in a closure to close reliably before we execute it
-	writeFile := func() error {
-		destFile, err := os.OpenFile(thisExecPath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, thisExecStat.Mode())
-		if err != nil {
-			return fmt.Errorf("unable to open destination file: %v", err)
-		}
-		defer destFile.Close()
-
-		l.Info("downloading binary", zap.String("source", urlStr), zap.String("destination", thisExecPath))
-
-		_, err = io.Copy(destFile, resp.Body)
-		if err != nil {
-			return fmt.Errorf("unable to download file: %v", err)
-		}
-
-		err = destFile.Sync()
-		if err != nil {
-			return fmt.Errorf("syncing downloaded file to device: %v", err)
-		}
-
-		return nil
-	}
-	err = writeFile()
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, err
-	}
-
-	l.Info("download successful; displaying new binary details", zap.String("location", thisExecPath))
-
-	// use the new binary to print out version and module info
-	fmt.Print("\nModule versions:\n\n")
-	cmd := exec.Command(thisExecPath, "list-modules", "--versions")
-	cmd.Stdout = os.Stdout
-	cmd.Stderr = os.Stderr
-	err = cmd.Run()
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to execute: %v", err)
-	}
-	fmt.Println("\nVersion:")
-	cmd = exec.Command(thisExecPath, "version")
-	cmd.Stdout = os.Stdout
-	cmd.Stderr = os.Stderr
-	err = cmd.Run()
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to execute: %v", err)
-	}
-	fmt.Println()
-
-	// clean up the backup file
-	err = os.Remove(backupExecPath)
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to clean up backup binary: %v", err)
-	}
-
-	l.Info("upgrade successful; please restart any running Caddy instances", zap.String("executable", thisExecPath))
-
-	return caddy.ExitCodeSuccess, nil
-}
-
 func cmdHelp(fl Flags) (int, error) {
 	const fullDocs = `Full documentation is available at:
 https://caddyserver.com/docs/command-line`
@@ -779,56 +640,6 @@ commands:
 	return caddy.ExitCodeSuccess, nil
 }
 
-func getModules() (standard, nonstandard, unknown []moduleInfo, err error) {
-	bi, ok := debug.ReadBuildInfo()
-	if !ok {
-		err = fmt.Errorf("no build info")
-		return
-	}
-
-	for _, modID := range caddy.Modules() {
-		modInfo, err := caddy.GetModule(modID)
-		if err != nil {
-			// that's weird, shouldn't happen
-			unknown = append(unknown, moduleInfo{caddyModuleID: modID, err: err})
-			continue
-		}
-
-		// to get the Caddy plugin's version info, we need to know
-		// the package that the Caddy module's value comes from; we
-		// can use reflection but we need a non-pointer value (I'm
-		// not sure why), and since New() should return a pointer
-		// value, we need to dereference it first
-		iface := interface{}(modInfo.New())
-		if rv := reflect.ValueOf(iface); rv.Kind() == reflect.Ptr {
-			iface = reflect.New(reflect.TypeOf(iface).Elem()).Elem().Interface()
-		}
-		modPkgPath := reflect.TypeOf(iface).PkgPath()
-
-		// now we find the Go module that the Caddy module's package
-		// belongs to; we assume the Caddy module package path will
-		// be prefixed by its Go module path, and we will choose the
-		// longest matching prefix in case there are nested modules
-		var matched *debug.Module
-		for _, dep := range bi.Deps {
-			if strings.HasPrefix(modPkgPath, dep.Path) {
-				if matched == nil || len(dep.Path) > len(matched.Path) {
-					matched = dep
-				}
-			}
-		}
-
-		caddyModGoMod := moduleInfo{caddyModuleID: modID, goModule: matched}
-
-		if strings.HasPrefix(modPkgPath, caddy.ImportPath) {
-			standard = append(standard, caddyModGoMod)
-		} else {
-			nonstandard = append(nonstandard, caddyModGoMod)
-		}
-	}
-	return
-}
-
 // apiRequest makes an API request to the endpoint adminAddr with the
 // given HTTP method and request URI. If body is non-nil, it will be
 // assumed to be Content-Type application/json.
@@ -895,7 +706,7 @@ func apiRequest(adminAddr, method, uri string, headers http.Header, body io.Read
 
 	// if it didn't work, let the user know
 	if resp.StatusCode >= 400 {
-		respBody, err := ioutil.ReadAll(io.LimitReader(resp.Body, 1024*10))
+		respBody, err := io.ReadAll(io.LimitReader(resp.Body, 1024*10))
 		if err != nil {
 			return fmt.Errorf("HTTP %d: reading error message: %v", resp.StatusCode, err)
 		}

cmd/commands.go

@@ -61,6 +61,12 @@ type Command struct {
 // any error that occurred.
 type CommandFunc func(Flags) (int, error)
 
+// Commands returns a list of commands initialised by
+// RegisterCommand
+func Commands() map[string]Command {
+	return commands
+}
+
 var commands = make(map[string]Command)
 
 func init() {
@@ -202,6 +208,7 @@ config file; otherwise the default is assumed.`,
 			fs := flag.NewFlagSet("list-modules", flag.ExitOnError)
 			fs.Bool("packages", false, "Print package paths")
 			fs.Bool("versions", false, "Print version information")
+			fs.Bool("skip-standard", false, "Skip printing standard modules")
 			return fs
 		}(),
 	})
@@ -253,7 +260,7 @@ Loads and provisions the provided config, but does not start running it.
 This reveals any errors with the configuration through the loading and
 provisioning stages.`,
 		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("load", flag.ExitOnError)
+			fs := flag.NewFlagSet("validate", flag.ExitOnError)
 			fs.String("config", "", "Input configuration file")
 			fs.String("adapter", "", "Name of config adapter")
 			return fs
@@ -276,7 +283,7 @@ If you wish you use stdin instead of a regular file, use - as the path.
 When reading from stdin, the --overwrite flag has no effect: the result
 is always printed to stdout.`,
 		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("format", flag.ExitOnError)
+			fs := flag.NewFlagSet("fmt", flag.ExitOnError)
 			fs.Bool("overwrite", false, "Overwrite the input file with the results")
 			return fs
 		}(),
@@ -289,6 +296,45 @@ is always printed to stdout.`,
 		Long: `
 Downloads an updated Caddy binary with the same modules/plugins at the
 latest versions. EXPERIMENTAL: May be changed or removed.`,
+		Flags: func() *flag.FlagSet {
+			fs := flag.NewFlagSet("upgrade", flag.ExitOnError)
+			fs.Bool("keep-backup", false, "Keep the backed up binary, instead of deleting it")
+			return fs
+		}(),
+	})
+
+	RegisterCommand(Command{
+		Name:  "add-package",
+		Func:  cmdAddPackage,
+		Usage: "<packages...>",
+		Short: "Adds Caddy packages (EXPERIMENTAL)",
+		Long: `
+Downloads an updated Caddy binary with the specified packages (module/plugin)
+added. Retains existing packages. Returns an error if the any of packages are 
+already included. EXPERIMENTAL: May be changed or removed.
+`,
+		Flags: func() *flag.FlagSet {
+			fs := flag.NewFlagSet("add-package", flag.ExitOnError)
+			fs.Bool("keep-backup", false, "Keep the backed up binary, instead of deleting it")
+			return fs
+		}(),
+	})
+
+	RegisterCommand(Command{
+		Name:  "remove-package",
+		Func:  cmdRemovePackage,
+		Usage: "<packages...>",
+		Short: "Removes Caddy packages (EXPERIMENTAL)",
+		Long: `
+Downloads an updated Caddy binaries without the specified packages (module/plugin). 
+Returns an error if any of the packages are not included. 
+EXPERIMENTAL: May be changed or removed.
+`,
+		Flags: func() *flag.FlagSet {
+			fs := flag.NewFlagSet("remove-package", flag.ExitOnError)
+			fs.Bool("keep-backup", false, "Keep the backed up binary, instead of deleting it")
+			return fs
+		}(),
 	})
 
 }

cmd/main.go

@@ -20,7 +20,6 @@ import (
 	"flag"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
 	"net"
 	"os"
@@ -94,7 +93,7 @@ func Main() {
 // the bytes in expect, or returns an error if it doesn't.
 func handlePingbackConn(conn net.Conn, expect []byte) error {
 	defer conn.Close()
-	confirmationBytes, err := ioutil.ReadAll(io.LimitReader(conn, 32))
+	confirmationBytes, err := io.ReadAll(io.LimitReader(conn, 32))
 	if err != nil {
 		return err
 	}
@@ -124,9 +123,9 @@ func loadConfig(configFile, adapterName string) ([]byte, string, error) {
 	var err error
 	if configFile != "" {
 		if configFile == "-" {
-			config, err = ioutil.ReadAll(os.Stdin)
+			config, err = io.ReadAll(os.Stdin)
 		} else {
-			config, err = ioutil.ReadFile(configFile)
+			config, err = os.ReadFile(configFile)
 		}
 		if err != nil {
 			return nil, "", fmt.Errorf("reading config file: %v", err)
@@ -140,7 +139,7 @@ func loadConfig(configFile, adapterName string) ([]byte, string, error) {
 		// plugged in, and if so, try using a default Caddyfile
 		cfgAdapter = caddyconfig.GetAdapter("caddyfile")
 		if cfgAdapter != nil {
-			config, err = ioutil.ReadFile("Caddyfile")
+			config, err = os.ReadFile("Caddyfile")
 			if os.IsNotExist(err) {
 				// okay, no default Caddyfile; pretend like this never happened
 				cfgAdapter = nil
@@ -361,6 +360,11 @@ func loadEnvFromFile(envFile string) error {
 		}
 	}
 
+	// Update the storage paths to ensure they have the proper
+	// value after loading a specified env file.
+	caddy.ConfigAutosavePath = filepath.Join(caddy.AppConfigDir(), "autosave.json")
+	caddy.DefaultStorage = &certmagic.FileStorage{Path: caddy.AppDataDir()}
+
 	return nil
 }
 
@@ -415,7 +419,7 @@ func printEnvironment() {
 	fmt.Printf("caddy.AppDataDir=%s\n", caddy.AppDataDir())
 	fmt.Printf("caddy.AppConfigDir=%s\n", caddy.AppConfigDir())
 	fmt.Printf("caddy.ConfigAutosavePath=%s\n", caddy.ConfigAutosavePath)
-	fmt.Printf("caddy.Version=%s\n", caddyVersion())
+	fmt.Printf("caddy.Version=%s\n", CaddyVersion())
 	fmt.Printf("runtime.GOOS=%s\n", runtime.GOOS)
 	fmt.Printf("runtime.GOARCH=%s\n", runtime.GOARCH)
 	fmt.Printf("runtime.Compiler=%s\n", runtime.Compiler)
@@ -432,8 +436,8 @@ func printEnvironment() {
 	}
 }
 
-// caddyVersion returns a detailed version string, if available.
-func caddyVersion() string {
+// CaddyVersion returns a detailed version string, if available.
+func CaddyVersion() string {
 	goModule := caddy.GoModule()
 	ver := goModule.Version
 	if goModule.Sum != "" {

cmd/packagesfuncs.go

@@ -0,0 +1,303 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddycmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"os/exec"
+	"reflect"
+	"runtime"
+	"runtime/debug"
+	"strings"
+
+	"github.com/caddyserver/caddy/v2"
+	"go.uber.org/zap"
+)
+
+func cmdUpgrade(fl Flags) (int, error) {
+	_, nonstandard, _, err := getModules()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("unable to enumerate installed plugins: %v", err)
+	}
+	pluginPkgs, err := getPluginPackages(nonstandard)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	return upgradeBuild(pluginPkgs, fl)
+}
+
+func cmdAddPackage(fl Flags) (int, error) {
+	if len(fl.Args()) == 0 {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("at least one package name must be specified")
+	}
+	_, nonstandard, _, err := getModules()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("unable to enumerate installed plugins: %v", err)
+	}
+	pluginPkgs, err := getPluginPackages(nonstandard)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	for _, arg := range fl.Args() {
+		if _, ok := pluginPkgs[arg]; ok {
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("package is already added")
+		}
+		pluginPkgs[arg] = struct{}{}
+	}
+
+	return upgradeBuild(pluginPkgs, fl)
+}
+
+func cmdRemovePackage(fl Flags) (int, error) {
+	if len(fl.Args()) == 0 {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("at least one package name must be specified")
+	}
+	_, nonstandard, _, err := getModules()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("unable to enumerate installed plugins: %v", err)
+	}
+	pluginPkgs, err := getPluginPackages(nonstandard)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	for _, arg := range fl.Args() {
+		if _, ok := pluginPkgs[arg]; !ok {
+			// package does not exist
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("package is not added")
+		}
+		delete(pluginPkgs, arg)
+	}
+
+	return upgradeBuild(pluginPkgs, fl)
+}
+
+func upgradeBuild(pluginPkgs map[string]struct{}, fl Flags) (int, error) {
+	l := caddy.Log()
+
+	thisExecPath, err := os.Executable()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("determining current executable path: %v", err)
+	}
+	thisExecStat, err := os.Stat(thisExecPath)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("retrieving current executable permission bits: %v", err)
+	}
+	l.Info("this executable will be replaced", zap.String("path", thisExecPath))
+
+	// build the request URL to download this custom build
+	qs := url.Values{
+		"os":   {runtime.GOOS},
+		"arch": {runtime.GOARCH},
+	}
+	for pkg := range pluginPkgs {
+		qs.Add("p", pkg)
+	}
+
+	// initiate the build
+	resp, err := downloadBuild(qs)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("download failed: %v", err)
+	}
+	defer resp.Body.Close()
+
+	// back up the current binary, in case something goes wrong we can replace it
+	backupExecPath := thisExecPath + ".tmp"
+	l.Info("build acquired; backing up current executable",
+		zap.String("current_path", thisExecPath),
+		zap.String("backup_path", backupExecPath))
+	err = os.Rename(thisExecPath, backupExecPath)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("backing up current binary: %v", err)
+	}
+	defer func() {
+		if err != nil {
+			err2 := os.Rename(backupExecPath, thisExecPath)
+			if err2 != nil {
+				l.Error("restoring original executable failed; will need to be restored manually",
+					zap.String("backup_path", backupExecPath),
+					zap.String("original_path", thisExecPath),
+					zap.Error(err2))
+			}
+		}
+	}()
+
+	// download the file; do this in a closure to close reliably before we execute it
+	err = writeCaddyBinary(thisExecPath, &resp.Body, thisExecStat)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	l.Info("download successful; displaying new binary details", zap.String("location", thisExecPath))
+
+	// use the new binary to print out version and module info
+	fmt.Print("\nModule versions:\n\n")
+	if err = listModules(thisExecPath); err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to execute 'caddy list-modules': %v", err)
+	}
+	fmt.Println("\nVersion:")
+	if err = showVersion(thisExecPath); err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to execute 'caddy version': %v", err)
+	}
+	fmt.Println()
+
+	// clean up the backup file
+	if !fl.Bool("keep-backup") {
+		if err = removeCaddyBinary(backupExecPath); err != nil {
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to clean up backup binary: %v", err)
+		}
+	} else {
+		l.Info("skipped cleaning up the backup file", zap.String("backup_path", backupExecPath))
+	}
+
+	l.Info("upgrade successful; please restart any running Caddy instances", zap.String("executable", thisExecPath))
+
+	return caddy.ExitCodeSuccess, nil
+}
+
+func getModules() (standard, nonstandard, unknown []moduleInfo, err error) {
+	bi, ok := debug.ReadBuildInfo()
+	if !ok {
+		err = fmt.Errorf("no build info")
+		return
+	}
+
+	for _, modID := range caddy.Modules() {
+		modInfo, err := caddy.GetModule(modID)
+		if err != nil {
+			// that's weird, shouldn't happen
+			unknown = append(unknown, moduleInfo{caddyModuleID: modID, err: err})
+			continue
+		}
+
+		// to get the Caddy plugin's version info, we need to know
+		// the package that the Caddy module's value comes from; we
+		// can use reflection but we need a non-pointer value (I'm
+		// not sure why), and since New() should return a pointer
+		// value, we need to dereference it first
+		iface := interface{}(modInfo.New())
+		if rv := reflect.ValueOf(iface); rv.Kind() == reflect.Ptr {
+			iface = reflect.New(reflect.TypeOf(iface).Elem()).Elem().Interface()
+		}
+		modPkgPath := reflect.TypeOf(iface).PkgPath()
+
+		// now we find the Go module that the Caddy module's package
+		// belongs to; we assume the Caddy module package path will
+		// be prefixed by its Go module path, and we will choose the
+		// longest matching prefix in case there are nested modules
+		var matched *debug.Module
+		for _, dep := range bi.Deps {
+			if strings.HasPrefix(modPkgPath, dep.Path) {
+				if matched == nil || len(dep.Path) > len(matched.Path) {
+					matched = dep
+				}
+			}
+		}
+
+		caddyModGoMod := moduleInfo{caddyModuleID: modID, goModule: matched}
+
+		if strings.HasPrefix(modPkgPath, caddy.ImportPath) {
+			standard = append(standard, caddyModGoMod)
+		} else {
+			nonstandard = append(nonstandard, caddyModGoMod)
+		}
+	}
+	return
+}
+
+func listModules(path string) error {
+	cmd := exec.Command(path, "list-modules", "--versions", "--skip-standard")
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	return cmd.Run()
+}
+
+func showVersion(path string) error {
+	cmd := exec.Command(path, "version")
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	return cmd.Run()
+}
+
+func downloadBuild(qs url.Values) (*http.Response, error) {
+	l := caddy.Log()
+	l.Info("requesting build",
+		zap.String("os", qs.Get("os")),
+		zap.String("arch", qs.Get("arch")),
+		zap.Strings("packages", qs["p"]))
+	resp, err := http.Get(fmt.Sprintf("%s?%s", downloadPath, qs.Encode()))
+	if err != nil {
+		return nil, fmt.Errorf("secure request failed: %v", err)
+	}
+	if resp.StatusCode >= 400 {
+		var details struct {
+			StatusCode int `json:"status_code"`
+			Error      struct {
+				Message string `json:"message"`
+				ID      string `json:"id"`
+			} `json:"error"`
+		}
+		err2 := json.NewDecoder(resp.Body).Decode(&details)
+		if err2 != nil {
+			return nil, fmt.Errorf("download and error decoding failed: HTTP %d: %v", resp.StatusCode, err2)
+		}
+		return nil, fmt.Errorf("download failed: HTTP %d: %s (id=%s)", resp.StatusCode, details.Error.Message, details.Error.ID)
+	}
+	return resp, nil
+}
+
+func getPluginPackages(modules []moduleInfo) (map[string]struct{}, error) {
+	pluginPkgs := make(map[string]struct{})
+	for _, mod := range modules {
+		if mod.goModule.Replace != nil {
+			return nil, fmt.Errorf("cannot auto-upgrade when Go module has been replaced: %s => %s",
+				mod.goModule.Path, mod.goModule.Replace.Path)
+		}
+		pluginPkgs[mod.goModule.Path] = struct{}{}
+	}
+	return pluginPkgs, nil
+}
+
+func writeCaddyBinary(path string, body *io.ReadCloser, fileInfo os.FileInfo) error {
+	l := caddy.Log()
+	destFile, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, fileInfo.Mode())
+	if err != nil {
+		return fmt.Errorf("unable to open destination file: %v", err)
+	}
+	defer destFile.Close()
+
+	l.Info("downloading binary", zap.String("destination", path))
+
+	_, err = io.Copy(destFile, *body)
+	if err != nil {
+		return fmt.Errorf("unable to download file: %v", err)
+	}
+
+	err = destFile.Sync()
+	if err != nil {
+		return fmt.Errorf("syncing downloaded file to device: %v", err)
+	}
+
+	return nil
+}
+
+const downloadPath = "https://caddyserver.com/api/download"

cmd/removebinary.go

@@ -0,0 +1,30 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !windows
+// +build !windows
+
+package caddycmd
+
+import (
+	"os"
+)
+
+// removeCaddyBinary removes the Caddy binary at the given path.
+//
+// On any non-Windows OS, this simply calls os.Remove, since they should
+// probably not exhibit any issue with processes deleting themselves.
+func removeCaddyBinary(path string) error {
+	return os.Remove(path)
+}

cmd/removebinary_windows.go

@@ -0,0 +1,36 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddycmd
+
+import (
+	"os"
+	"path/filepath"
+	"syscall"
+)
+
+// removeCaddyBinary removes the Caddy binary at the given path.
+//
+// On Windows, this uses a syscall to indirectly remove the file,
+// because otherwise we get an "Access is denied." error when trying
+// to delete the binary while Caddy is still running and performing
+// the upgrade. "cmd.exe /C" executes a command specified by the
+// following arguments, i.e. "del" which will run as a separate process,
+// which avoids the "Access is denied." error.
+func removeCaddyBinary(path string) error {
+	var sI syscall.StartupInfo
+	var pI syscall.ProcessInformation
+	argv := syscall.StringToUTF16Ptr(filepath.Join(os.Getenv("windir"), "system32", "cmd.exe") + " /C del " + path)
+	return syscall.CreateProcess(nil, argv, nil, nil, true, 0, nil, nil, &sI, &pI)
+}

go.mod

@@ -1,35 +1,39 @@
 module github.com/caddyserver/caddy/v2
 
-go 1.15
+go 1.16
 
 require (
-	github.com/Masterminds/sprig/v3 v3.1.0
-	github.com/alecthomas/chroma v0.8.2
-	github.com/aryann/difflib v0.0.0-20170710044230-e206f873d14a
-	github.com/caddyserver/certmagic v0.13.1
+	github.com/Masterminds/sprig/v3 v3.2.2
+	github.com/alecthomas/chroma v0.9.2
+	github.com/aryann/difflib v0.0.0-20210328193216-ff5ff6dc229b
+	github.com/caddyserver/certmagic v0.15.2
 	github.com/dustin/go-humanize v1.0.1-0.20200219035652-afde56e7acac
 	github.com/go-chi/chi v4.1.2+incompatible
-	github.com/google/cel-go v0.6.0
-	github.com/google/uuid v1.2.0
-	github.com/klauspost/compress v1.11.3
-	github.com/klauspost/cpuid/v2 v2.0.6
-	github.com/lucas-clemente/quic-go v0.20.1
-	github.com/mholt/acmez v0.1.3
+	github.com/google/cel-go v0.7.3
+	github.com/google/uuid v1.3.0
+	github.com/klauspost/compress v1.13.6
+	github.com/klauspost/cpuid/v2 v2.0.9
+	github.com/lucas-clemente/quic-go v0.23.0
+	github.com/mholt/acmez v1.0.1
 	github.com/naoina/go-stringutil v0.1.0 // indirect
 	github.com/naoina/toml v0.1.1
-	github.com/prometheus/client_golang v1.9.0
-	github.com/smallstep/certificates v0.15.4
-	github.com/smallstep/cli v0.15.2
-	github.com/smallstep/nosql v0.3.0 // cannot upgrade from v0.3.0 until protobuf warning is fixed
+	github.com/prometheus/client_golang v1.11.0
+	github.com/smallstep/certificates v0.17.5-0.20211008195551-04fe3126bebf
+	github.com/smallstep/cli v0.17.6
+	github.com/smallstep/nosql v0.3.8
 	github.com/smallstep/truststore v0.9.6
-	github.com/yuin/goldmark v1.2.1
-	github.com/yuin/goldmark-highlighting v0.0.0-20200307114337-60d527fdb691
-	go.uber.org/zap v1.16.0
-	golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de
-	golang.org/x/net v0.0.0-20201110031124-69a78807bb2b
-	golang.org/x/term v0.0.0-20210317153231-de623e64d2a6
-	google.golang.org/genproto v0.0.0-20200806141610-86f49bd18e98
-	google.golang.org/protobuf v1.24.0 // cannot upgrade until warning is fixed
+	github.com/yuin/goldmark v1.4.1
+	github.com/yuin/goldmark-highlighting v0.0.0-20210516132338-9216f9c5aa01
+	go.uber.org/zap v1.19.0
+	golang.org/x/crypto v0.0.0-20210915214749-c084706c2272
+	golang.org/x/net v0.0.0-20210913180222-943fd674d43e
+	golang.org/x/term v0.0.0-20210503060354-a79de5458b56
+	google.golang.org/genproto v0.0.0-20210719143636-1d5a45f8e492
+	google.golang.org/protobuf v1.27.1
 	gopkg.in/natefinch/lumberjack.v2 v2.0.0
-	gopkg.in/yaml.v2 v2.3.0
+	gopkg.in/yaml.v2 v2.4.0
 )
+
+// avoid license conflict from juju/ansiterm until https://github.com/manifoldco/promptui/pull/181
+// is merged or other dependency in path currently in violation fixes compliance
+replace github.com/manifoldco/promptui => github.com/nguyer/promptui v0.8.1-0.20210517132806-70ccd4709797

logging.go

@@ -18,7 +18,6 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
 	"os"
 	"strings"
@@ -630,9 +629,9 @@ func (StderrWriter) OpenWriter() (io.WriteCloser, error) {
 	return notClosable{os.Stderr}, nil
 }
 
-// OpenWriter returns ioutil.Discard that can't be closed.
+// OpenWriter returns io.Discard that can't be closed.
 func (DiscardWriter) OpenWriter() (io.WriteCloser, error) {
-	return notClosable{ioutil.Discard}, nil
+	return notClosable{io.Discard}, nil
 }
 
 // notClosable is an io.WriteCloser that can't be closed.

metrics.go

@@ -6,12 +6,13 @@ import (
 	"strings"
 
 	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/collectors"
 	"github.com/prometheus/client_golang/prometheus/promauto"
 )
 
 // define and register the metrics used in this package.
 func init() {
-	prometheus.MustRegister(prometheus.NewBuildInfoCollector())
+	prometheus.MustRegister(collectors.NewBuildInfoCollector())
 
 	const ns, sub = "caddy", "admin"
 

modules/caddyhttp/app.go

@@ -77,6 +77,7 @@ func init() {
 // `{http.request.tls.client.public_key}` | The public key of the client certificate.
 // `{http.request.tls.client.public_key_sha256}` | The SHA256 checksum of the client's public key.
 // `{http.request.tls.client.certificate_pem}` | The PEM-encoded value of the certificate.
+// `{http.request.tls.client.certificate_der_base64}` | The base64-encoded value of the certificate.
 // `{http.request.tls.client.issuer}` | The issuer DN of the client certificate
 // `{http.request.tls.client.serial}` | The serial number of the client certificate
 // `{http.request.tls.client.subject}` | The subject DN of the client certificate

modules/caddyhttp/caddyhttp.go

@@ -20,7 +20,10 @@ import (
 	"io"
 	"net"
 	"net/http"
+	"net/url"
+	"path/filepath"
 	"strconv"
+	"strings"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
@@ -217,6 +220,32 @@ func StatusCodeMatches(actual, configured int) bool {
 	return false
 }
 
+// SanitizedPathJoin performs filepath.Join(root, reqPath) that
+// is safe against directory traversal attacks. It uses logic
+// similar to that in the Go standard library, specifically
+// in the implementation of http.Dir. The root is assumed to
+// be a trusted path, but reqPath is not; and the output will
+// never be outside of root. The resulting path can be used
+// with the local file system.
+func SanitizedPathJoin(root, reqPath string) string {
+	reqPath, _ = url.PathUnescape(reqPath)
+	if root == "" {
+		root = "."
+	}
+
+	path := filepath.Join(root, filepath.Clean("/"+reqPath))
+
+	// filepath.Join also cleans the path, and cleaning strips
+	// the trailing slash, so we need to re-add it afterwards.
+	// if the length is 1, then it's a path to the root,
+	// and that should return ".", so we don't append the separator.
+	if strings.HasSuffix(reqPath, "/") && len(reqPath) > 1 {
+		path += separator
+	}
+
+	return path
+}
+
 // tlsPlaceholderWrapper is a no-op listener wrapper that marks
 // where the TLS listener should be in a chain of listener wrappers.
 // It should only be used if another listener wrapper must be placed
@@ -242,6 +271,8 @@ const (
 	DefaultHTTPSPort = 443
 )
 
+const separator = string(filepath.Separator)
+
 // Interface guard
 var _ caddy.ListenerWrapper = (*tlsPlaceholderWrapper)(nil)
 var _ caddyfile.Unmarshaler = (*tlsPlaceholderWrapper)(nil)

modules/caddyhttp/caddyhttp_test.go

@@ -0,0 +1,94 @@
+package caddyhttp
+
+import (
+	"net/url"
+	"path/filepath"
+	"testing"
+)
+
+func TestSanitizedPathJoin(t *testing.T) {
+	// For reference:
+	// %2e = .
+	// %2f = /
+	// %5c = \
+	for i, tc := range []struct {
+		inputRoot string
+		inputPath string
+		expect    string
+	}{
+		{
+			inputPath: "",
+			expect:    ".",
+		},
+		{
+			inputPath: "/",
+			expect:    ".",
+		},
+		{
+			inputPath: "/foo",
+			expect:    "foo",
+		},
+		{
+			inputPath: "/foo/",
+			expect:    "foo" + separator,
+		},
+		{
+			inputPath: "/foo/bar",
+			expect:    filepath.Join("foo", "bar"),
+		},
+		{
+			inputRoot: "/a",
+			inputPath: "/foo/bar",
+			expect:    filepath.Join("/", "a", "foo", "bar"),
+		},
+		{
+			inputPath: "/foo/../bar",
+			expect:    "bar",
+		},
+		{
+			inputRoot: "/a/b",
+			inputPath: "/foo/../bar",
+			expect:    filepath.Join("/", "a", "b", "bar"),
+		},
+		{
+			inputRoot: "/a/b",
+			inputPath: "/..%2fbar",
+			expect:    filepath.Join("/", "a", "b", "bar"),
+		},
+		{
+			inputRoot: "/a/b",
+			inputPath: "/%2e%2e%2fbar",
+			expect:    filepath.Join("/", "a", "b", "bar"),
+		},
+		{
+			inputRoot: "/a/b",
+			inputPath: "/%2e%2e%2f%2e%2e%2f",
+			expect:    filepath.Join("/", "a", "b") + separator,
+		},
+		{
+			inputRoot: "C:\\www",
+			inputPath: "/foo/bar",
+			expect:    filepath.Join("C:\\www", "foo", "bar"),
+		},
+		{
+			inputRoot: "C:\\www",
+			inputPath: "/D:\\foo\\bar",
+			expect:    filepath.Join("C:\\www", "D:\\foo\\bar"),
+		},
+	} {
+		// we don't *need* to use an actual parsed URL, but it
+		// adds some authenticity to the tests since real-world
+		// values will be coming in from URLs; thus, the test
+		// corpus can contain paths as encoded by clients, which
+		// more closely emulates the actual attack vector
+		u, err := url.Parse("http://test:9999" + tc.inputPath)
+		if err != nil {
+			t.Fatalf("Test %d: invalid URL: %v", i, err)
+		}
+		actual := SanitizedPathJoin(tc.inputRoot, u.Path)
+		if actual != tc.expect {
+			t.Errorf("Test %d: SanitizedPathJoin('%s', '%s') =>  %s (expected '%s')",
+				i, tc.inputRoot, tc.inputPath, actual, tc.expect)
+		}
+	}
+}

modules/caddyhttp/celmatcher.go

@@ -35,7 +35,6 @@ import (
 	"github.com/google/cel-go/interpreter/functions"
 	exprpb "google.golang.org/genproto/googleapis/api/expr/v1alpha1"
 	"google.golang.org/protobuf/proto"
-	timestamp "google.golang.org/protobuf/types/known/timestamppb"
 )
 
 func init() {
@@ -231,8 +230,7 @@ func (celTypeAdapter) NativeToValue(value interface{}) ref.Val {
 	case pkix.Name:
 		return celPkixName{&v}
 	case time.Time:
-		// TODO: eliminate direct protobuf dependency, sigh -- just wrap stdlib time.Time instead...
-		return types.Timestamp{Timestamp: &timestamp.Timestamp{Seconds: v.Unix(), Nanos: int32(v.Nanosecond())}}
+		return types.Timestamp{Time: v}
 	case error:
 		types.NewErr(v.Error())
 	}

modules/caddyhttp/encode/encode.go

@@ -182,6 +182,7 @@ type responseWriter struct {
 	buf          *bytes.Buffer
 	config       *Encode
 	statusCode   int
+	wroteHeader  bool
 }
 
 // WriteHeader stores the status to write when the time comes
@@ -195,6 +196,19 @@ func (enc *Encode) Match(rw *responseWriter) bool {
 	return enc.Matcher.Match(rw.statusCode, rw.Header())
 }
 
+// Flush implements http.Flusher. It delays the actual Flush of the underlying ResponseWriterWrapper
+// until headers were written.
+func (rw *responseWriter) Flush() {
+	if !rw.wroteHeader {
+		// flushing the underlying ResponseWriter will write header and status code,
+		// but we need to delay that until we can determine if we must encode and
+		// therefore add the Content-Encoding header; this happens in the first call
+		// to rw.Write (see bug in #4314)
+		return
+	}
+	rw.ResponseWriterWrapper.Flush()
+}
+
 // Write writes to the response. If the response qualifies,
 // it is encoded using the encoder, which is initialized
 // if not done so already.
@@ -225,6 +239,7 @@ func (rw *responseWriter) Write(p []byte) (int, error) {
 	if rw.statusCode > 0 {
 		rw.ResponseWriter.WriteHeader(rw.statusCode)
 		rw.statusCode = 0
+		rw.wroteHeader = true
 	}
 
 	switch {
@@ -271,6 +286,7 @@ func (rw *responseWriter) Close() error {
 		// that rely on If-None-Match, for example
 		rw.ResponseWriter.WriteHeader(rw.statusCode)
 		rw.statusCode = 0
+		rw.wroteHeader = true
 	}
 	if rw.w != nil {
 		err2 := rw.w.Close()

modules/caddyhttp/encode/gzip/gzip.go

@@ -15,7 +15,6 @@
 package caddygzip
 
 import (
-	"compress/flate"
 	"fmt"
 	"strconv"
 
@@ -68,11 +67,11 @@ func (g *Gzip) Provision(ctx caddy.Context) error {
 
 // Validate validates g's configuration.
 func (g Gzip) Validate() error {
-	if g.Level < flate.NoCompression {
-		return fmt.Errorf("quality too low; must be >= %d", flate.NoCompression)
+	if g.Level < gzip.StatelessCompression {
+		return fmt.Errorf("quality too low; must be >= %d", gzip.StatelessCompression)
 	}
-	if g.Level > flate.BestCompression {
-		return fmt.Errorf("quality too high; must be <= %d", flate.BestCompression)
+	if g.Level > gzip.BestCompression {
+		return fmt.Errorf("quality too high; must be <= %d", gzip.BestCompression)
 	}
 	return nil
 }

modules/caddyhttp/encode/zstd/zstd.go

@@ -47,7 +47,10 @@ func (Zstd) AcceptEncoding() string { return "zstd" }
 
 // NewEncoder returns a new gzip writer.
 func (z Zstd) NewEncoder() encode.Encoder {
-	writer, _ := zstd.NewWriter(nil)
+	// The default of 8MB for the window is
+	// too large for many clients, so we limit
+	// it to 128K to lighten their load.
+	writer, _ := zstd.NewWriter(nil, zstd.WithWindowSize(128<<10), zstd.WithEncoderConcurrency(1), zstd.WithZeroFrames(true))
 	return writer
 }
 

modules/caddyhttp/fileserver/browse.go

@@ -22,6 +22,7 @@ import (
 	"os"
 	"path"
 	"strings"
+	"sync"
 	"text/template"
 
 	"github.com/caddyserver/caddy/v2"
@@ -34,8 +35,6 @@ import (
 type Browse struct {
 	// Use this template file instead of the default browse template.
 	TemplateFile string `json:"template_file,omitempty"`
-
-	template *template.Template
 }
 
 func (fsrv *FileServer) serveBrowse(root, dirPath string, w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
@@ -43,15 +42,28 @@ func (fsrv *FileServer) serveBrowse(root, dirPath string, w http.ResponseWriter,
 		zap.String("path", dirPath),
 		zap.String("root", root))
 
-	// navigation on the client-side gets messed up if the
-	// URL doesn't end in a trailing slash because hrefs like
-	// "/b/c" on a path like "/a" end up going to "/b/c" instead
+	// Navigation on the client-side gets messed up if the
+	// URL doesn't end in a trailing slash because hrefs to
+	// "b/c" at path "/a" end up going to "/b/c" instead
 	// of "/a/b/c" - so we have to redirect in this case
-	if !strings.HasSuffix(r.URL.Path, "/") {
-		fsrv.logger.Debug("redirecting to trailing slash to preserve hrefs", zap.String("request_path", r.URL.Path))
-		r.URL.Path += "/"
-		http.Redirect(w, r, r.URL.String(), http.StatusMovedPermanently)
-		return nil
+	// so that the path is "/a/" and the client constructs
+	// relative hrefs "b/c" to be "/a/b/c".
+	//
+	// Only redirect if the last element of the path (the filename) was not
+	// rewritten; if the admin wanted to rewrite to the canonical path, they
+	// would have, and we have to be very careful not to introduce unwanted
+	// redirects and especially redirect loops! (Redirecting using the
+	// original URI is necessary because that's the URI the browser knows,
+	// we don't want to redirect from internally-rewritten URIs.)
+	// See https://github.com/caddyserver/caddy/issues/4205.
+	origReq := r.Context().Value(caddyhttp.OriginalRequestCtxKey).(http.Request)
+	if path.Base(origReq.URL.Path) == path.Base(r.URL.Path) {
+		if !strings.HasSuffix(origReq.URL.Path, "/") {
+			fsrv.logger.Debug("redirecting to trailing slash to preserve hrefs", zap.String("request_path", r.URL.Path))
+			origReq.URL.Path += "/"
+			http.Redirect(w, r, origReq.URL.String(), http.StatusMovedPermanently)
+			return nil
+		}
 	}
 
 	dir, err := fsrv.openFile(dirPath, w)
@@ -75,11 +87,14 @@ func (fsrv *FileServer) serveBrowse(root, dirPath string, w http.ResponseWriter,
 
 	fsrv.browseApplyQueryParams(w, r, &listing)
 
-	// write response as either JSON or HTML
-	var buf *bytes.Buffer
+	buf := bufPool.Get().(*bytes.Buffer)
+	defer bufPool.Put(buf)
+
 	acceptHeader := strings.ToLower(strings.Join(r.Header["Accept"], ","))
+
+	// write response as either JSON or HTML
 	if strings.Contains(acceptHeader, "application/json") {
-		if buf, err = fsrv.browseWriteJSON(listing); err != nil {
+		if err := json.NewEncoder(buf).Encode(listing.Items); err != nil {
 			return caddyhttp.Error(http.StatusInternalServerError, err)
 		}
 		w.Header().Set("Content-Type", "application/json; charset=utf-8")
@@ -98,12 +113,11 @@ func (fsrv *FileServer) serveBrowse(root, dirPath string, w http.ResponseWriter,
 			browseTemplateContext: listing,
 		}
 
-		err = fsrv.makeBrowseTemplate(tplCtx)
+		tpl, err := fsrv.makeBrowseTemplate(tplCtx)
 		if err != nil {
 			return fmt.Errorf("parsing browse template: %v", err)
 		}
-
-		if buf, err = fsrv.browseWriteHTML(tplCtx); err != nil {
+		if err := tpl.Execute(buf, tplCtx); err != nil {
 			return caddyhttp.Error(http.StatusInternalServerError, err)
 		}
 		w.Header().Set("Content-Type", "text/html; charset=utf-8")
@@ -123,7 +137,12 @@ func (fsrv *FileServer) loadDirectoryContents(dir *os.File, root, urlPath string
 	// user can presumably browse "up" to parent folder if path is longer than "/"
 	canGoUp := len(urlPath) > 1
 
-	return fsrv.directoryListing(files, canGoUp, root, urlPath, repl), nil
+	l, err := fsrv.directoryListing(files, canGoUp, root, urlPath, repl)
+	if err != nil {
+		return browseTemplateContext{}, err
+	}
+
+	return l, nil
 }
 
 // browseApplyQueryParams applies query parameters to the listing.
@@ -161,7 +180,7 @@ func (fsrv *FileServer) browseApplyQueryParams(w http.ResponseWriter, r *http.Re
 }
 
 // makeBrowseTemplate creates the template to be used for directory listings.
-func (fsrv *FileServer) makeBrowseTemplate(tplCtx *templateContext) error {
+func (fsrv *FileServer) makeBrowseTemplate(tplCtx *templateContext) (*template.Template, error) {
 	var tpl *template.Template
 	var err error
 
@@ -169,33 +188,17 @@ func (fsrv *FileServer) makeBrowseTemplate(tplCtx *templateContext) error {
 		tpl = tplCtx.NewTemplate(path.Base(fsrv.Browse.TemplateFile))
 		tpl, err = tpl.ParseFiles(fsrv.Browse.TemplateFile)
 		if err != nil {
-			return fmt.Errorf("parsing browse template file: %v", err)
+			return nil, fmt.Errorf("parsing browse template file: %v", err)
 		}
 	} else {
 		tpl = tplCtx.NewTemplate("default_listing")
 		tpl, err = tpl.Parse(defaultBrowseTemplate)
 		if err != nil {
-			return fmt.Errorf("parsing default browse template: %v", err)
+			return nil, fmt.Errorf("parsing default browse template: %v", err)
 		}
 	}
 
-	fsrv.Browse.template = tpl
-
-	return nil
-}
-
-func (fsrv *FileServer) browseWriteJSON(listing browseTemplateContext) (*bytes.Buffer, error) {
-	buf := bufPool.Get().(*bytes.Buffer)
-	defer bufPool.Put(buf)
-	err := json.NewEncoder(buf).Encode(listing.Items)
-	return buf, err
-}
-
-func (fsrv *FileServer) browseWriteHTML(tplCtx *templateContext) (*bytes.Buffer, error) {
-	buf := bufPool.Get().(*bytes.Buffer)
-	defer bufPool.Put(buf)
-	err := fsrv.Browse.template.Execute(buf, tplCtx)
-	return buf, err
+	return tpl, nil
 }
 
 // isSymlink return true if f is a symbolic link
@@ -209,7 +212,7 @@ func isSymlinkTargetDir(f os.FileInfo, root, urlPath string) bool {
 	if !isSymlink(f) {
 		return false
 	}
-	target := sanitizedPathJoin(root, path.Join(urlPath, f.Name()))
+	target := caddyhttp.SanitizedPathJoin(root, path.Join(urlPath, f.Name()))
 	targetInfo, err := os.Stat(target)
 	if err != nil {
 		return false
@@ -224,3 +227,10 @@ type templateContext struct {
 	templates.TemplateContext
 	browseTemplateContext
 }
+
+// bufPool is used to increase the efficiency of file listings.
+var bufPool = sync.Pool{
+	New: func() interface{} {
+		return new(bytes.Buffer)
+	},
+}

modules/caddyhttp/fileserver/browse_test.go

@@ -1,68 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fileserver
-
-import (
-	"testing"
-	"text/template"
-)
-
-func BenchmarkBrowseWriteJSON(b *testing.B) {
-	fsrv := new(FileServer)
-	listing := browseTemplateContext{
-		Name:     "test",
-		Path:     "test",
-		CanGoUp:  false,
-		Items:    make([]fileInfo, 100),
-		NumDirs:  42,
-		NumFiles: 420,
-		Sort:     "",
-		Order:    "",
-		Limit:    42,
-	}
-	b.ResetTimer()
-
-	for n := 0; n < b.N; n++ {
-		fsrv.browseWriteJSON(listing)
-	}
-}
-
-func BenchmarkBrowseWriteHTML(b *testing.B) {
-	fsrv := new(FileServer)
-	fsrv.Browse = &Browse{
-		TemplateFile: "",
-		template:     template.New("test"),
-	}
-	listing := browseTemplateContext{
-		Name:     "test",
-		Path:     "test",
-		CanGoUp:  false,
-		Items:    make([]fileInfo, 100),
-		NumDirs:  42,
-		NumFiles: 420,
-		Sort:     "",
-		Order:    "",
-		Limit:    42,
-	}
-	tplCtx := &templateContext{
-		browseTemplateContext: listing,
-	}
-	fsrv.makeBrowseTemplate(tplCtx)
-	b.ResetTimer()
-
-	for n := 0; n < b.N; n++ {
-		fsrv.browseWriteHTML(tplCtx)
-	}
-}

modules/caddyhttp/fileserver/browsetpl.go

@@ -39,6 +39,10 @@ h1 a:hover {
 	color: #319cff;
 }
 
+a:visited {
+  color: #800080;
+}
+
 header,
 #summary {
 	padding-left: 5%;
@@ -426,7 +430,7 @@ footer {
 		</footer>
 		<script>
 			var filterEl = document.getElementById('filter');
-			filterEl.focus();
+			filterEl.focus({ preventScroll: true });
 
 			function initFilter() {
 				if (!filterEl.value) {

modules/caddyhttp/fileserver/browsetplcontext.go

@@ -27,7 +27,7 @@ import (
 	"github.com/dustin/go-humanize"
 )
 
-func (fsrv *FileServer) directoryListing(files []os.FileInfo, canGoUp bool, root, urlPath string, repl *caddy.Replacer) browseTemplateContext {
+func (fsrv *FileServer) directoryListing(files []os.FileInfo, canGoUp bool, root, urlPath string, repl *caddy.Replacer) (browseTemplateContext, error) {
 	filesToHide := fsrv.transformHidePaths(repl)
 
 	var dirCount, fileCount int
@@ -42,20 +42,31 @@ func (fsrv *FileServer) directoryListing(files []os.FileInfo, canGoUp bool, root
 
 		isDir := f.IsDir() || isSymlinkTargetDir(f, root, urlPath)
 
+		u := url.URL{Path: url.PathEscape(name)}
+
+		// add the slash after the escape of path to avoid escaping the slash as well
 		if isDir {
-			name += "/"
+			u.Path += "/"
 			dirCount++
 		} else {
 			fileCount++
 		}
 
-		u := url.URL{Path: "./" + name} // prepend with "./" to fix paths with ':' in the name
+		fileIsSymlink := isSymlink(f)
+		size := f.Size()
+		if fileIsSymlink {
+			info, err := os.Stat(name)
+			if err != nil {
+				return browseTemplateContext{}, err
+			}
+			size = info.Size()
+		}
 
 		fileInfos = append(fileInfos, fileInfo{
 			IsDir:     isDir,
-			IsSymlink: isSymlink(f),
-			Name:      f.Name(),
-			Size:      f.Size(),
+			IsSymlink: fileIsSymlink,
+			Name:      name,
+			Size:      size,
 			URL:       u.String(),
 			ModTime:   f.ModTime().UTC(),
 			Mode:      f.Mode(),
@@ -69,7 +80,7 @@ func (fsrv *FileServer) directoryListing(files []os.FileInfo, canGoUp bool, root
 		Items:    fileInfos,
 		NumDirs:  dirCount,
 		NumFiles: fileCount,
-	}
+	}, nil
 }
 
 // browseTemplateContext provides the template context for directory listings.
@@ -264,7 +275,7 @@ func (l byTime) Less(i, j int) bool { return l.Items[i].ModTime.Before(l.Items[j
 
 const (
 	sortByName         = "name"
-	sortByNameDirFirst = "name_dir_first"
+	sortByNameDirFirst = "namedirfirst"
 	sortBySize         = "size"
 	sortByTime         = "time"
 )

modules/caddyhttp/fileserver/caddyfile.go

@@ -41,6 +41,7 @@ func init() {
 //        browse        [<template_file>]
 //        precompressed <formats...>
 //        status        <status>
+//        disable_canonical_uris
 //    }
 //
 func parseCaddyfile(h httpcaddyfile.Helper) (caddyhttp.MiddlewareHandler, error) {
@@ -112,6 +113,13 @@ func parseCaddyfile(h httpcaddyfile.Helper) (caddyhttp.MiddlewareHandler, error)
 				}
 				fsrv.StatusCode = caddyhttp.WeakString(h.Val())
 
+			case "disable_canonical_uris":
+				if h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				falseBool := false
+				fsrv.CanonicalURIs = &falseBool
+
 			default:
 				return nil, h.Errf("unknown subdirective '%s'", h.Val())
 			}

modules/caddyhttp/fileserver/matcher.go

@@ -19,6 +19,7 @@ import (
 	"net/http"
 	"os"
 	"path"
+	"strconv"
 	"strings"
 	"time"
 
@@ -60,7 +61,11 @@ type MatchFile struct {
 	// directories are treated distinctly, so to match
 	// a directory, the filepath MUST end in a forward
 	// slash `/`. To match a regular file, there must
-	// be no trailing slash. Accepts placeholders.
+	// be no trailing slash. Accepts placeholders. If
+	// the policy is "first_exist", then an error may
+	// be triggered as a fallback by configuring "="
+	// followed by a status code number,
+	// for example "=404".
 	TryFiles []string `json:"try_files,omitempty"`
 
 	// How to choose a file in TryFiles. Can be:
@@ -185,7 +190,7 @@ func (m MatchFile) selectFile(r *http.Request) (matched bool) {
 		if strings.HasSuffix(file, "/") {
 			suffix += "/"
 		}
-		fullpath = sanitizedPathJoin(root, suffix)
+		fullpath = caddyhttp.SanitizedPathJoin(root, suffix)
 		return
 	}
 
@@ -205,6 +210,10 @@ func (m MatchFile) selectFile(r *http.Request) (matched bool) {
 	switch m.TryPolicy {
 	case "", tryPolicyFirstExist:
 		for _, f := range m.TryFiles {
+			if err := parseErrorCode(f); err != nil {
+				caddyhttp.SetVar(r.Context(), caddyhttp.MatcherErrorVarKey, err)
+				return
+			}
 			suffix, fullpath, remainder := prepareFilePath(f)
 			if info, exists := strictFileExists(fullpath); exists {
 				setPlaceholders(info, suffix, fullpath, remainder)
@@ -274,6 +283,20 @@ func (m MatchFile) selectFile(r *http.Request) (matched bool) {
 	return
 }
 
+// parseErrorCode checks if the input is a status
+// code number, prefixed by "=", and returns an
+// error if so.
+func parseErrorCode(input string) error {
+	if len(input) > 1 && input[0] == '=' {
+		code, err := strconv.Atoi(input[1:])
+		if err != nil || code < 100 || code > 999 {
+			return nil
+		}
+		return caddyhttp.Error(code, fmt.Errorf("%s", input[1:]))
+	}
+	return nil
+}
+
 // strictFileExists returns true if file exists
 // and matches the convention of the given file
 // path. If the path ends in a forward slash,

modules/caddyhttp/fileserver/matcher_test.go

@@ -17,12 +17,31 @@ package fileserver
 import (
 	"net/http"
 	"net/url"
+	"os"
+	"runtime"
 	"testing"
 
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 )
 
 func TestFileMatcher(t *testing.T) {
+
+	// Windows doesn't like colons in files names
+	isWindows := runtime.GOOS == "windows"
+	if !isWindows {
+		filename := "with:in-name.txt"
+		f, err := os.Create("./testdata/" + filename)
+		if err != nil {
+			t.Fail()
+			return
+		}
+		t.Cleanup(func() {
+			os.Remove("./testdata/" + filename)
+		})
+		f.WriteString(filename)
+		f.Close()
+	}
+
 	for i, tc := range []struct {
 		path         string
 		expectedPath string
@@ -63,6 +82,30 @@ func TestFileMatcher(t *testing.T) {
 			path:    "/missingfile.php",
 			matched: false,
 		},
+		{
+			path:         "ملف.txt", // the path file name is not escaped
+			expectedPath: "ملف.txt",
+			expectedType: "file",
+			matched:      true,
+		},
+		{
+			path:         url.PathEscape("ملف.txt"), // singly-escaped path
+			expectedPath: "ملف.txt",
+			expectedType: "file",
+			matched:      true,
+		},
+		{
+			path:         url.PathEscape(url.PathEscape("ملف.txt")), // doubly-escaped path
+			expectedPath: "%D9%85%D9%84%D9%81.txt",
+			expectedType: "file",
+			matched:      true,
+		},
+		{
+			path:         "./with:in-name.txt", // browsers send the request with the path as such
+			expectedPath: "with:in-name.txt",
+			expectedType: "file",
+			matched:      !isWindows,
+		},
 	} {
 		m := &MatchFile{
 			Root:     "./testdata",
@@ -94,7 +137,7 @@ func TestFileMatcher(t *testing.T) {
 			t.Fatalf("Test %d: actual path: %v, expected: %v", i, rel, tc.expectedPath)
 		}
 
-		fileType, ok := repl.Get("http.matchers.file.type")
+		fileType, _ := repl.Get("http.matchers.file.type")
 		if fileType != tc.expectedType {
 			t.Fatalf("Test %d: actual file type: %v, expected: %v", i, fileType, tc.expectedType)
 		}
@@ -197,7 +240,7 @@ func TestPHPFileMatcher(t *testing.T) {
 			t.Fatalf("Test %d: actual path: %v, expected: %v", i, rel, tc.expectedPath)
 		}
 
-		fileType, ok := repl.Get("http.matchers.file.type")
+		fileType, _ := repl.Get("http.matchers.file.type")
 		if fileType != tc.expectedType {
 			t.Fatalf("Test %d: actual file type: %v, expected: %v", i, fileType, tc.expectedType)
 		}

modules/caddyhttp/fileserver/staticfiles.go

@@ -15,16 +15,16 @@
 package fileserver
 
 import (
-	"bytes"
 	"fmt"
 	weakrand "math/rand"
 	"mime"
 	"net/http"
+	"net/url"
 	"os"
+	"path"
 	"path/filepath"
 	"strconv"
 	"strings"
-	"sync"
 	"time"
 
 	"github.com/caddyserver/caddy/v2"
@@ -71,6 +71,10 @@ type FileServer struct {
 
 	// Use redirects to enforce trailing slashes for directories, or to
 	// remove trailing slash from URIs for files. Default is true.
+	//
+	// Canonicalization will not happen if the last element of the request's
+	// path (the filename) is changed in an internal rewrite, to avoid
+	// clobbering the explicit rewrite with implicit behavior.
 	CanonicalURIs *bool `json:"canonical_uris,omitempty"`
 
 	// Override the status code written when successfully serving a file.
@@ -162,7 +166,17 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 	filesToHide := fsrv.transformHidePaths(repl)
 
 	root := repl.ReplaceAll(fsrv.Root, ".")
-	filename := sanitizedPathJoin(root, r.URL.Path)
+	// PathUnescape returns an error if the escapes aren't well-formed,
+	// meaning the count % matches the RFC. Return early if the escape is
+	// improper.
+	if _, err := url.PathUnescape(r.URL.Path); err != nil {
+		fsrv.logger.Debug("improper path escape",
+			zap.String("site_root", root),
+			zap.String("request_path", r.URL.Path),
+			zap.Error(err))
+		return err
+	}
+	filename := caddyhttp.SanitizedPathJoin(root, r.URL.Path)
 
 	fsrv.logger.Debug("sanitized path join",
 		zap.String("site_root", root),
@@ -178,7 +192,6 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 		} else if os.IsPermission(err) {
 			return caddyhttp.Error(http.StatusForbidden, err)
 		}
-		// TODO: treat this as resource exhaustion like with os.Open? Or unnecessary here?
 		return caddyhttp.Error(http.StatusInternalServerError, err)
 	}
 
@@ -187,7 +200,7 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 	var implicitIndexFile bool
 	if info.IsDir() && len(fsrv.IndexNames) > 0 {
 		for _, indexPage := range fsrv.IndexNames {
-			indexPath := sanitizedPathJoin(filename, indexPage)
+			indexPath := caddyhttp.SanitizedPathJoin(filename, indexPage)
 			if fileHidden(indexPath, filesToHide) {
 				// pretend this file doesn't exist
 				fsrv.logger.Debug("hiding index file",
@@ -243,12 +256,26 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 	// trailing slash - not enforcing this can break relative hrefs
 	// in HTML (see https://github.com/caddyserver/caddy/issues/2741)
 	if fsrv.CanonicalURIs == nil || *fsrv.CanonicalURIs {
-		if implicitIndexFile && !strings.HasSuffix(r.URL.Path, "/") {
-			fsrv.logger.Debug("redirecting to canonical URI (adding trailing slash for directory)", zap.String("path", r.URL.Path))
-			return redirect(w, r, r.URL.Path+"/")
-		} else if !implicitIndexFile && strings.HasSuffix(r.URL.Path, "/") {
-			fsrv.logger.Debug("redirecting to canonical URI (removing trailing slash for file)", zap.String("path", r.URL.Path))
-			return redirect(w, r, r.URL.Path[:len(r.URL.Path)-1])
+		// Only redirect if the last element of the path (the filename) was not
+		// rewritten; if the admin wanted to rewrite to the canonical path, they
+		// would have, and we have to be very careful not to introduce unwanted
+		// redirects and especially redirect loops!
+		// See https://github.com/caddyserver/caddy/issues/4205.
+		origReq := r.Context().Value(caddyhttp.OriginalRequestCtxKey).(http.Request)
+		if path.Base(origReq.URL.Path) == path.Base(r.URL.Path) {
+			if implicitIndexFile && !strings.HasSuffix(origReq.URL.Path, "/") {
+				to := origReq.URL.Path + "/"
+				fsrv.logger.Debug("redirecting to canonical URI (adding trailing slash for directory)",
+					zap.String("from_path", origReq.URL.Path),
+					zap.String("to_path", to))
+				return redirect(w, r, to)
+			} else if !implicitIndexFile && strings.HasSuffix(origReq.URL.Path, "/") {
+				to := origReq.URL.Path[:len(origReq.URL.Path)-1]
+				fsrv.logger.Debug("redirecting to canonical URI (removing trailing slash for file)",
+					zap.String("from_path", origReq.URL.Path),
+					zap.String("to_path", to))
+				return redirect(w, r, to)
+			}
 		}
 	}
 
@@ -423,42 +450,6 @@ func (fsrv *FileServer) transformHidePaths(repl *caddy.Replacer) []string {
 	return hide
 }
 
-// sanitizedPathJoin performs filepath.Join(root, reqPath) that
-// is safe against directory traversal attacks. It uses logic
-// similar to that in the Go standard library, specifically
-// in the implementation of http.Dir. The root is assumed to
-// be a trusted path, but reqPath is not.
-func sanitizedPathJoin(root, reqPath string) string {
-	// TODO: Caddy 1 uses this:
-	// prevent absolute path access on Windows, e.g. http://localhost:5000/C:\Windows\notepad.exe
-	// if runtime.GOOS == "windows" && len(reqPath) > 0 && filepath.IsAbs(reqPath[1:]) {
-	// TODO.
-	// }
-
-	// TODO: whereas std lib's http.Dir.Open() uses this:
-	// if filepath.Separator != '/' && strings.ContainsRune(name, filepath.Separator) {
-	// 	return nil, errors.New("http: invalid character in file path")
-	// }
-
-	// TODO: see https://play.golang.org/p/oh77BiVQFti for another thing to consider
-
-	if root == "" {
-		root = "."
-	}
-
-	path := filepath.Join(root, filepath.Clean("/"+reqPath))
-
-	// filepath.Join also cleans the path, and cleaning strips
-	// the trailing slash, so we need to re-add it afterwards.
-	// if the length is 1, then it's a path to the root,
-	// and that should return ".", so we don't append the separator.
-	if strings.HasSuffix(reqPath, "/") && len(reqPath) > 1 {
-		path += separator
-	}
-
-	return path
-}
-
 // fileHidden returns true if filename is hidden according to the hide list.
 // filename must be a relative or absolute file system path, not a request
 // URI path. It is expected that all the paths in the hide list are absolute
@@ -555,12 +546,6 @@ func (wr statusOverrideResponseWriter) WriteHeader(int) {
 
 var defaultIndexNames = []string{"index.html", "index.txt"}
 
-var bufPool = sync.Pool{
-	New: func() interface{} {
-		return new(bytes.Buffer)
-	},
-}
-
 const (
 	minBackoff, maxBackoff = 2, 5
 	separator              = string(filepath.Separator)

modules/caddyhttp/fileserver/staticfiles_test.go

@@ -15,96 +15,12 @@
 package fileserver
 
 import (
-	"net/url"
 	"path/filepath"
 	"runtime"
 	"strings"
 	"testing"
 )
 
-func TestSanitizedPathJoin(t *testing.T) {
-	// For easy reference:
-	// %2e = .
-	// %2f = /
-	// %5c = \
-	for i, tc := range []struct {
-		inputRoot string
-		inputPath string
-		expect    string
-	}{
-		{
-			inputPath: "",
-			expect:    ".",
-		},
-		{
-			inputPath: "/",
-			expect:    ".",
-		},
-		{
-			inputPath: "/foo",
-			expect:    "foo",
-		},
-		{
-			inputPath: "/foo/",
-			expect:    "foo" + separator,
-		},
-		{
-			inputPath: "/foo/bar",
-			expect:    filepath.Join("foo", "bar"),
-		},
-		{
-			inputRoot: "/a",
-			inputPath: "/foo/bar",
-			expect:    filepath.Join("/", "a", "foo", "bar"),
-		},
-		{
-			inputPath: "/foo/../bar",
-			expect:    "bar",
-		},
-		{
-			inputRoot: "/a/b",
-			inputPath: "/foo/../bar",
-			expect:    filepath.Join("/", "a", "b", "bar"),
-		},
-		{
-			inputRoot: "/a/b",
-			inputPath: "/..%2fbar",
-			expect:    filepath.Join("/", "a", "b", "bar"),
-		},
-		{
-			inputRoot: "/a/b",
-			inputPath: "/%2e%2e%2fbar",
-			expect:    filepath.Join("/", "a", "b", "bar"),
-		},
-		{
-			inputRoot: "/a/b",
-			inputPath: "/%2e%2e%2f%2e%2e%2f",
-			expect:    filepath.Join("/", "a", "b") + separator,
-		},
-		{
-			inputRoot: "C:\\www",
-			inputPath: "/foo/bar",
-			expect:    filepath.Join("C:\\www", "foo", "bar"),
-		},
-		// TODO: test more windows paths... on windows... sigh.
-	} {
-		// we don't *need* to use an actual parsed URL, but it
-		// adds some authenticity to the tests since real-world
-		// values will be coming in from URLs; thus, the test
-		// corpus can contain paths as encoded by clients, which
-		// more closely emulates the actual attack vector
-		u, err := url.Parse("http://test:9999" + tc.inputPath)
-		if err != nil {
-			t.Fatalf("Test %d: invalid URL: %v", i, err)
-		}
-		actual := sanitizedPathJoin(tc.inputRoot, u.Path)
-		if actual != tc.expect {
-			t.Errorf("Test %d: [%s %s] => %s (expected %s)",
-				i, tc.inputRoot, tc.inputPath, actual, tc.expect)
-		}
-	}
-}
-
 func TestFileHidden(t *testing.T) {
 	for i, tc := range []struct {
 		inputHide []string

modules/caddyhttp/fileserver/testdata/%D9%85%D9%84%D9%81.txt

@@ -0,0 +1 @@
+%D9%85%D9%84%D9%81.txt

modules/caddyhttp/fileserver/testdata/ملف.txt

@@ -0,0 +1 @@
+ملف.txt

modules/caddyhttp/headers/headers.go

@@ -213,7 +213,7 @@ func (ops HeaderOps) ApplyTo(hdr http.Header, repl *caddy.Replacer) {
 
 	// replace
 	for fieldName, replacements := range ops.Replace {
-		fieldName = repl.ReplaceAll(fieldName, "")
+		fieldName = http.CanonicalHeaderKey(repl.ReplaceAll(fieldName, ""))
 
 		// all fields...
 		if fieldName == "*" {
@@ -237,11 +237,17 @@ func (ops HeaderOps) ApplyTo(hdr http.Header, repl *caddy.Replacer) {
 		for _, r := range replacements {
 			search := repl.ReplaceAll(r.Search, "")
 			replace := repl.ReplaceAll(r.Replace, "")
-			for i := range hdr[fieldName] {
-				if r.re != nil {
-					hdr[fieldName][i] = r.re.ReplaceAllString(hdr[fieldName][i], replace)
-				} else {
-					hdr[fieldName][i] = strings.ReplaceAll(hdr[fieldName][i], search, replace)
+			for hdrFieldName, vals := range hdr {
+				// see issue #4330 for why we don't simply use hdr[fieldName]
+				if http.CanonicalHeaderKey(hdrFieldName) != fieldName {
+					continue
+				}
+				for i := range vals {
+					if r.re != nil {
+						hdr[hdrFieldName][i] = r.re.ReplaceAllString(hdr[hdrFieldName][i], replace)
+					} else {
+						hdr[hdrFieldName][i] = strings.ReplaceAll(hdr[hdrFieldName][i], search, replace)
+					}
 				}
 			}
 		}

modules/caddyhttp/headers/headers_test.go

@@ -160,6 +160,28 @@ func TestHandler(t *testing.T) {
 				"Fail-5xx": []string{"true"},
 			},
 		},
+		{
+			handler: Handler{
+				Request: &HeaderOps{
+					Replace: map[string][]Replacement{
+						"Case-Insensitive": {
+							Replacement{
+								Search:  "issue4330",
+								Replace: "issue #4330",
+							},
+						},
+					},
+				},
+			},
+			reqHeader: http.Header{
+				"case-insensitive": []string{"issue4330"},
+				"Other-Header":     []string{"issue4330"},
+			},
+			expectedReqHeader: http.Header{
+				"case-insensitive": []string{"issue #4330"},
+				"Other-Header":     []string{"issue4330"},
+			},
+		},
 	} {
 		rr := httptest.NewRecorder()
 
@@ -191,7 +213,7 @@ func TestHandler(t *testing.T) {
 		})
 
 		if err := tc.handler.ServeHTTP(rr, req, next); err != nil {
-			t.Errorf("Test %d: %w", i, err)
+			t.Errorf("Test %d: %v", i, err)
 			continue
 		}
 

modules/caddyhttp/map/map.go

@@ -131,23 +131,21 @@ func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhtt
 		// find the first mapping matching the input and return
 		// the requested destination/output value
 		for _, m := range h.Mappings {
-			if m.re != nil {
-				if m.re.MatchString(input) {
-					if output := m.Outputs[destIdx]; output == nil {
-						continue
-					} else {
-						output = m.re.ReplaceAllString(input, m.Outputs[destIdx].(string))
-						return output, true
-					}
-				}
+			output := m.Outputs[destIdx]
+			if output == nil {
 				continue
 			}
-			if input == m.Input {
-				if output := m.Outputs[destIdx]; output == nil {
+			if m.re != nil {
+				var result []byte
+				matches := m.re.FindStringSubmatchIndex(input)
+				if matches == nil {
 					continue
-				} else {
-					return output, true
 				}
+				result = m.re.ExpandString(result, output.(string), input, matches)
+				return string(result), true
+			}
+			if input == m.Input {
+				return output, true
 			}
 		}
 

modules/caddyhttp/map/map_test.go

@@ -0,0 +1,113 @@
+package maphandler
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"reflect"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
+)
+
+func TestHandler(t *testing.T) {
+	for i, tc := range []struct {
+		handler Handler
+		reqURI  string
+		expect  map[string]interface{}
+	}{
+		{
+			reqURI: "/foo",
+			handler: Handler{
+				Source:       "{http.request.uri.path}",
+				Destinations: []string{"{output}"},
+				Mappings: []Mapping{
+					{
+						Input:   "/foo",
+						Outputs: []interface{}{"FOO"},
+					},
+				},
+			},
+			expect: map[string]interface{}{
+				"output": "FOO",
+			},
+		},
+		{
+			reqURI: "/abcdef",
+			handler: Handler{
+				Source:       "{http.request.uri.path}",
+				Destinations: []string{"{output}"},
+				Mappings: []Mapping{
+					{
+						InputRegexp: "(/abc)",
+						Outputs:     []interface{}{"ABC"},
+					},
+				},
+			},
+			expect: map[string]interface{}{
+				"output": "ABC",
+			},
+		},
+		{
+			reqURI: "/ABCxyzDEF",
+			handler: Handler{
+				Source:       "{http.request.uri.path}",
+				Destinations: []string{"{output}"},
+				Mappings: []Mapping{
+					{
+						InputRegexp: "(xyz)",
+						Outputs:     []interface{}{"...${1}..."},
+					},
+				},
+			},
+			expect: map[string]interface{}{
+				"output": "...xyz...",
+			},
+		},
+		{
+			// Test case from https://caddy.community/t/map-directive-and-regular-expressions/13866/14?u=matt
+			reqURI: "/?s=0%27+AND+%28SELECT+0+FROM+%28SELECT+count%28%2A%29%2C+CONCAT%28%28SELECT+%40%40version%29%2C+0x23%2C+FLOOR%28RAND%280%29%2A2%29%29+AS+x+FROM+information_schema.columns+GROUP+BY+x%29+y%29+-+-+%27",
+			handler: Handler{
+				Source:       "{http.request.uri}",
+				Destinations: []string{"{output}"},
+				Mappings: []Mapping{
+					{
+						InputRegexp: "(?i)(\\^|`|<|>|%|\\\\|\\{|\\}|\\|)",
+						Outputs:     []interface{}{"3"},
+					},
+				},
+			},
+			expect: map[string]interface{}{
+				"output": "3",
+			},
+		},
+	} {
+		if err := tc.handler.Provision(caddy.Context{}); err != nil {
+			t.Fatalf("Test %d: Provisioning handler: %v", i, err)
+		}
+
+		req, err := http.NewRequest(http.MethodGet, tc.reqURI, nil)
+		if err != nil {
+			t.Fatalf("Test %d: Creating request: %v", i, err)
+		}
+		repl := caddyhttp.NewTestReplacer(req)
+		ctx := context.WithValue(req.Context(), caddy.ReplacerCtxKey, repl)
+		req = req.WithContext(ctx)
+
+		rr := httptest.NewRecorder()
+		noop := caddyhttp.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) error { return nil })
+
+		if err := tc.handler.ServeHTTP(rr, req, noop); err != nil {
+			t.Errorf("Test %d: Handler returned error: %v", i, err)
+			continue
+		}
+
+		for key, expected := range tc.expect {
+			actual, _ := repl.Get(key)
+			if !reflect.DeepEqual(actual, expected) {
+				t.Errorf("Test %d: Expected %#v but got %#v for {%s}", i, expected, actual, key)
+			}
+		}
+	}
+}

modules/caddyhttp/matchers.go

@@ -21,6 +21,7 @@ import (
 	"net/http"
 	"net/textproto"
 	"net/url"
+	"path"
 	"path/filepath"
 	"regexp"
 	"sort"
@@ -82,10 +83,23 @@ type (
 	// MatchMethod matches requests by the method.
 	MatchMethod []string
 
-	// MatchQuery matches requests by URI's query string.
+	// MatchQuery matches requests by the URI's query string. It takes a JSON object
+	// keyed by the query keys, with an array of string values to match for that key.
+	// Query key matches are exact, but wildcards may be used for value matches. Both
+	// keys and values may be placeholders.
+	// An example of the structure to match `?key=value&topic=api&query=something` is:
+	//
+	// ```json
+	// {
+	// 	"key": ["value"],
+	//	"topic": ["api"],
+	//	"query": ["*"]
+	// }
+	// ```
 	MatchQuery url.Values
 
-	// MatchHeader matches requests by header fields. It performs fast,
+	// MatchHeader matches requests by header fields. The key is the field
+	// name and the array is the list of field values. It performs fast,
 	// exact string comparisons of the field values. Fast prefix, suffix,
 	// and substring matches can also be done by suffixing, prefixing, or
 	// surrounding the value with the wildcard `*` character, respectively.
@@ -102,7 +116,8 @@ type (
 	// (potentially leading to collisions).
 	MatchHeaderRE map[string]*MatchRegexp
 
-	// MatchProtocol matches requests by protocol.
+	// MatchProtocol matches requests by protocol. Recognized values are
+	// "http", "https", and "grpc".
 	MatchProtocol string
 
 	// MatchRemoteIP matches requests by client IP (or CIDR range).
@@ -127,9 +142,9 @@ type (
 	// matchers within a set work the same (i.e. different matchers in
 	// the same set are AND'ed).
 	//
-	// Note that the generated docs which describe the structure of
-	// this module are wrong because of how this type unmarshals JSON
-	// in a custom way. The correct structure is:
+	// NOTE: The generated docs which describe the structure of this
+	// module are wrong because of how this type unmarshals JSON in a
+	// custom way. The correct structure is:
 	//
 	// ```json
 	// [
@@ -300,7 +315,15 @@ func (m MatchPath) Provision(_ caddy.Context) error {
 
 // Match returns true if r matches m.
 func (m MatchPath) Match(r *http.Request) bool {
-	lowerPath := strings.ToLower(r.URL.Path)
+	// PathUnescape returns an error if the escapes aren't
+	// well-formed, meaning the count % matches the RFC.
+	// Return early if the escape is improper.
+	unescapedPath, err := url.PathUnescape(r.URL.Path)
+	if err != nil {
+		return false
+	}
+
+	lowerPath := strings.ToLower(unescapedPath)
 
 	// see #2917; Windows ignores trailing dots and spaces
 	// when accessing files (sigh), potentially causing a
@@ -309,6 +332,16 @@ func (m MatchPath) Match(r *http.Request) bool {
 	// being matched by *.php to be treated as PHP scripts
 	lowerPath = strings.TrimRight(lowerPath, ". ")
 
+	// Clean the path, merges doubled slashes, etc.
+	// This ensures maliciously crafted requests can't bypass
+	// the path matcher. See #4407
+	lowerPath = path.Clean(lowerPath)
+
+	// Cleaning may remove the trailing slash, but we want to keep it
+	if lowerPath != "/" && strings.HasSuffix(r.URL.Path, "/") {
+		lowerPath = lowerPath + "/"
+	}
+
 	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
 
 	for _, matchPath := range m {
@@ -382,7 +415,26 @@ func (MatchPathRE) CaddyModule() caddy.ModuleInfo {
 // Match returns true if r matches m.
 func (m MatchPathRE) Match(r *http.Request) bool {
 	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
-	return m.MatchRegexp.Match(r.URL.Path, repl)
+
+	// PathUnescape returns an error if the escapes aren't
+	// well-formed, meaning the count % matches the RFC.
+	// Return early if the escape is improper.
+	unescapedPath, err := url.PathUnescape(r.URL.Path)
+	if err != nil {
+		return false
+	}
+
+	// Clean the path, merges doubled slashes, etc.
+	// This ensures maliciously crafted requests can't bypass
+	// the path matcher. See #4407
+	cleanedPath := path.Clean(unescapedPath)
+
+	// Cleaning may remove the trailing slash, but we want to keep it
+	if cleanedPath != "/" && strings.HasSuffix(r.URL.Path, "/") {
+		cleanedPath = cleanedPath + "/"
+	}
+
+	return m.MatchRegexp.Match(cleanedPath, repl)
 }
 
 // CaddyModule returns the Caddy module information.
@@ -667,7 +719,7 @@ func (MatchProtocol) CaddyModule() caddy.ModuleInfo {
 func (m MatchProtocol) Match(r *http.Request) bool {
 	switch string(m) {
 	case "grpc":
-		return r.Header.Get("content-type") == "application/grpc"
+		return strings.HasPrefix(r.Header.Get("content-type"), "application/grpc")
 	case "https":
 		return r.TLS != nil
 	case "http":
@@ -975,6 +1027,12 @@ var wordRE = regexp.MustCompile(`\w+`)
 
 const regexpPlaceholderPrefix = "http.regexp"
 
+// MatcherErrorVarKey is the key used for the variable that
+// holds an optional error emitted from a request matcher,
+// to short-circuit the handler chain, since matchers cannot
+// return errors via the RequestMatcher interface.
+const MatcherErrorVarKey = "matchers.error"
+
 // Interface guards
 var (
 	_ RequestMatcher    = (*MatchHost)(nil)

modules/caddyhttp/matchers_test.go

@@ -257,6 +257,21 @@ func TestPathMatcher(t *testing.T) {
 			input:  "/foo/BAR.txt",
 			expect: true,
 		},
+		{
+			match:  MatchPath{"/foo*"},
+			input:  "//foo/bar",
+			expect: true,
+		},
+		{
+			match:  MatchPath{"/foo*"},
+			input:  "//foo",
+			expect: true,
+		},
+		{
+			match:  MatchPath{"/foo*"},
+			input:  "/%2F/foo",
+			expect: true,
+		},
 		{
 			match:  MatchPath{"*"},
 			input:  "/",
@@ -326,15 +341,30 @@ func TestPathREMatcher(t *testing.T) {
 			expect: true,
 		},
 		{
-			match:  MatchPathRE{MatchRegexp{Pattern: "/foo"}},
+			match:  MatchPathRE{MatchRegexp{Pattern: "^/foo"}},
 			input:  "/foo",
 			expect: true,
 		},
 		{
-			match:  MatchPathRE{MatchRegexp{Pattern: "/foo"}},
+			match:  MatchPathRE{MatchRegexp{Pattern: "^/foo"}},
 			input:  "/foo/",
 			expect: true,
 		},
+		{
+			match:  MatchPathRE{MatchRegexp{Pattern: "^/foo"}},
+			input:  "//foo",
+			expect: true,
+		},
+		{
+			match:  MatchPathRE{MatchRegexp{Pattern: "^/foo"}},
+			input:  "//foo/",
+			expect: true,
+		},
+		{
+			match:  MatchPathRE{MatchRegexp{Pattern: "^/foo"}},
+			input:  "/%2F/foo/",
+			expect: true,
+		},
 		{
 			match:  MatchPathRE{MatchRegexp{Pattern: "/bar"}},
 			input:  "/foo/",

modules/caddyhttp/metrics_test.go

@@ -55,7 +55,7 @@ func TestMetricsInstrumentedHandler(t *testing.T) {
 
 	handlerErr = nil
 	if err := ih.ServeHTTP(w, r, h); err != nil {
-		t.Errorf("Received unexpected error: %w", err)
+		t.Errorf("Received unexpected error: %v", err)
 	}
 
 	// an empty handler - no errors, no header written
@@ -67,7 +67,7 @@ func TestMetricsInstrumentedHandler(t *testing.T) {
 	w = httptest.NewRecorder()
 
 	if err := ih.ServeHTTP(w, r, h); err != nil {
-		t.Errorf("Received unexpected error: %w", err)
+		t.Errorf("Received unexpected error: %v", err)
 	}
 	if actual := w.Result().StatusCode; actual != 200 {
 		t.Errorf("Not same: expected status code %#v, but got %#v", 200, actual)

modules/caddyhttp/replacer.go

@@ -25,10 +25,10 @@ import (
 	"crypto/tls"
 	"crypto/x509"
 	"encoding/asn1"
+	"encoding/base64"
 	"encoding/pem"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"net"
 	"net/http"
 	"net/textproto"
@@ -162,7 +162,7 @@ func addHTTPVarsToReplacer(repl *caddy.Replacer, req *http.Request, w http.Respo
 					return "", true
 				}
 				// replace real body with buffered data
-				req.Body = ioutil.NopCloser(buf)
+				req.Body = io.NopCloser(buf)
 				return buf.String(), true
 
 				// original request, before any internal changes
@@ -353,6 +353,8 @@ func getReqTLSReplacement(req *http.Request, key string) (interface{}, bool) {
 		case "client.certificate_pem":
 			block := pem.Block{Type: "CERTIFICATE", Bytes: cert.Raw}
 			return pem.EncodeToMemory(&block), true
+		case "client.certificate_der_base64":
+			return base64.StdEncoding.EncodeToString(cert.Raw), true
 		default:
 			return nil, false
 		}

modules/caddyhttp/responsematchers.go

@@ -32,7 +32,7 @@ type ResponseMatcher struct {
 
 	// If set, each header specified must be one of the
 	// specified values, with the same logic used by the
-	// request header matcher.
+	// [request header matcher](/docs/json/apps/http/servers/routes/match/header/).
 	Headers http.Header `json:"headers,omitempty"`
 }
 

modules/caddyhttp/reverseproxy/caddyfile.go

@@ -61,7 +61,7 @@ func parseCaddyfile(h httpcaddyfile.Helper) (caddyhttp.MiddlewareHandler, error)
 //         lb_try_interval <interval>
 //
 //         # active health checking
-//         health_path <path>
+//         health_uri  <uri>
 //         health_port <port>
 //         health_interval <interval>
 //         health_timeout <duration>
@@ -978,6 +978,18 @@ func (h *HTTPTransport) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 					h.KeepAlive = new(KeepAlive)
 				}
 				h.KeepAlive.MaxIdleConns = num
+
+			case "keepalive_idle_conns_per_host":
+				if !d.NextArg() {
+					return d.ArgErr()
+				}
+				num, err := strconv.Atoi(d.Val())
+				if err != nil {
+					return d.Errf("bad integer value '%s': %v", d.Val(), err)
+				}
+				if h.KeepAlive == nil {
+					h.KeepAlive = new(KeepAlive)
+				}
 				h.KeepAlive.MaxIdleConnsPerHost = num
 
 			case "versions":
@@ -1004,16 +1016,6 @@ func (h *HTTPTransport) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 				}
 				h.MaxConnsPerHost = num
 
-			case "max_idle_conns_per_host":
-				if !d.NextArg() {
-					return d.ArgErr()
-				}
-				num, err := strconv.Atoi(d.Val())
-				if err != nil {
-					return d.Errf("bad integer value '%s': %v", d.Val(), err)
-				}
-				h.MaxIdleConnsPerHost = num
-
 			default:
 				return d.Errf("unrecognized subdirective %s", d.Val())
 			}

modules/caddyhttp/reverseproxy/command.go

@@ -59,7 +59,7 @@ default, all incoming headers are passed through unmodified.)
 			fs.String("from", "localhost", "Address on which to receive traffic")
 			fs.String("to", "", "Upstream address to which to to proxy traffic")
 			fs.Bool("change-host-header", false, "Set upstream Host header to address of upstream")
-			fs.Bool("insecure", false, "Disable TLS verification (WARNING: DISABLES SECURITY, WHY ARE YOU EVEN USING TLS?)")
+			fs.Bool("insecure", false, "Disable TLS verification (WARNING: DISABLES SECURITY BY NOT VERIFYING SSL CERTIFICATES!)")
 			return fs
 		}(),
 	})

modules/caddyhttp/reverseproxy/fastcgi/caddyfile.go

@@ -124,21 +124,22 @@ func (t *Transport) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 //
 // is equivalent to a route consisting of:
 //
+//     # Add trailing slash for directory requests
 //     @canonicalPath {
-//         file {
-//             try_files {path}/index.php
-//         }
-//         not {
-//             path */
-//         }
+//         file {path}/index.php
+//         not path */
 //     }
 //     redir @canonicalPath {path}/ 308
 //
-//     try_files {path} {path}/index.php index.php
-//
-//     @phpFiles {
-//         path *.php
+//     # If the requested file does not exist, try index files
+//     @indexFiles file {
+//         try_files {path} {path}/index.php index.php
+//         split_path .php
 //     }
+//     rewrite @indexFiles {http.matchers.file.relative}
+//
+//     # Proxy PHP files to the FastCGI responder
+//     @phpFiles path *.php
 //     reverse_proxy @phpFiles localhost:7777 {
 //         transport fastcgi {
 //             split .php
@@ -172,6 +173,9 @@ func parsePHPFastCGI(h httpcaddyfile.Helper) ([]httpcaddyfile.ConfigValue, error
 	// set the default index file for the try_files rewrites
 	indexFile := "index.php"
 
+	// set up for explicitly overriding try_files
+	tryFiles := []string{}
+
 	// if the user specified a matcher token, use that
 	// matcher in a route that wraps both of our routes;
 	// either way, strip the matcher token and pass
@@ -192,7 +196,7 @@ func parsePHPFastCGI(h httpcaddyfile.Helper) ([]httpcaddyfile.ConfigValue, error
 	// NOTE: we delete the tokens as we go so that the reverse_proxy
 	// unmarshal doesn't see these subdirectives which it cannot handle
 	for dispenser.Next() {
-		for dispenser.NextBlock(0) {
+		for dispenser.NextBlock(0) && dispenser.Nesting() == 1 {
 			switch dispenser.Val() {
 			case "root":
 				if !dispenser.NextArg() {
@@ -237,6 +241,17 @@ func parsePHPFastCGI(h httpcaddyfile.Helper) ([]httpcaddyfile.ConfigValue, error
 				}
 				indexFile = args[0]
 
+			case "try_files":
+				args := dispenser.RemainingArgs()
+				dispenser.Delete()
+				for range args {
+					dispenser.Delete()
+				}
+				if len(args) < 1 {
+					return nil, dispenser.ArgErr()
+				}
+				tryFiles = args
+
 			case "resolve_root_symlink":
 				args := dispenser.RemainingArgs()
 				dispenser.Delete()
@@ -318,10 +333,15 @@ func parsePHPFastCGI(h httpcaddyfile.Helper) ([]httpcaddyfile.ConfigValue, error
 			HandlersRaw:    []json.RawMessage{caddyconfig.JSONModuleObject(redirHandler, "handler", "static_response", nil)},
 		}
 
+		// if tryFiles wasn't overridden, use a reasonable default
+		if len(tryFiles) == 0 {
+			tryFiles = []string{"{http.request.uri.path}", "{http.request.uri.path}/" + indexFile, indexFile}
+		}
+
 		// route to rewrite to PHP index file
 		rewriteMatcherSet := caddy.ModuleMap{
 			"file": h.JSON(fileserver.MatchFile{
-				TryFiles:  []string{"{http.request.uri.path}", "{http.request.uri.path}/" + indexFile, indexFile},
+				TryFiles:  tryFiles,
 				SplitPath: extensions,
 			}),
 		}

modules/caddyhttp/reverseproxy/fastcgi/client.go

@@ -30,7 +30,6 @@ import (
 	"encoding/binary"
 	"errors"
 	"io"
-	"io/ioutil"
 	"mime/multipart"
 	"net"
 	"net/http"
@@ -445,7 +444,7 @@ func (c *FCGIClient) Request(p map[string]string, req io.Reader) (resp *http.Res
 	if chunked(resp.TransferEncoding) {
 		resp.Body = clientCloser{c, httputil.NewChunkedReader(rb)}
 	} else {
-		resp.Body = clientCloser{c, ioutil.NopCloser(rb)}
+		resp.Body = clientCloser{c, io.NopCloser(rb)}
 	}
 	return
 }

modules/caddyhttp/reverseproxy/fastcgi/client_test.go

@@ -26,7 +26,6 @@ import (
 	"encoding/binary"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
 	"math/rand"
 	"net"
@@ -166,7 +165,7 @@ func sendFcgi(reqType int, fcgiParams map[string]string, data []byte, posts map[
 	}
 
 	defer resp.Body.Close()
-	content, _ = ioutil.ReadAll(resp.Body)
+	content, _ = io.ReadAll(resp.Body)
 
 	log.Println("c: send data length ≈", length, string(content))
 	fcgi.Close()

modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go

@@ -20,7 +20,6 @@ import (
 	"fmt"
 	"net"
 	"net/http"
-	"path"
 	"path/filepath"
 	"strconv"
 	"strings"
@@ -218,12 +217,7 @@ func (t Transport) buildEnv(r *http.Request) (map[string]string, error) {
 	}
 
 	// SCRIPT_FILENAME is the absolute path of SCRIPT_NAME
-	scriptFilename := filepath.Join(root, scriptName)
-
-	// Add vhost path prefix to scriptName. Otherwise, some PHP software will
-	// have difficulty discovering its URL.
-	pathPrefix, _ := r.Context().Value(caddy.CtxKey("path_prefix")).(string)
-	scriptName = path.Join(pathPrefix, scriptName)
+	scriptFilename := caddyhttp.SanitizedPathJoin(root, scriptName)
 
 	// Ensure the SCRIPT_NAME has a leading slash for compliance with RFC3875
 	// Info: https://tools.ietf.org/html/rfc3875#section-4.1.13
@@ -236,13 +230,7 @@ func (t Transport) buildEnv(r *http.Request) (map[string]string, error) {
 	// original URI in as the value of REQUEST_URI (the user can overwrite this
 	// if desired). Most PHP apps seem to want the original URI. Besides, this is
 	// how nginx defaults: http://stackoverflow.com/a/12485156/1048862
-	origReq, ok := r.Context().Value(caddyhttp.OriginalRequestCtxKey).(http.Request)
-	if !ok {
-		// some requests, like active health checks, don't add this to
-		// the request context, so we can just use the current URL
-		origReq = *r
-	}
-	reqURL := origReq.URL
+	origReq := r.Context().Value(caddyhttp.OriginalRequestCtxKey).(http.Request)
 
 	requestScheme := "http"
 	if r.TLS != nil {
@@ -285,7 +273,7 @@ func (t Transport) buildEnv(r *http.Request) (map[string]string, error) {
 		"DOCUMENT_ROOT":   root,
 		"DOCUMENT_URI":    docURI,
 		"HTTP_HOST":       r.Host, // added here, since not always part of headers
-		"REQUEST_URI":     reqURL.RequestURI(),
+		"REQUEST_URI":     origReq.URL.RequestURI(),
 		"SCRIPT_FILENAME": scriptFilename,
 		"SCRIPT_NAME":     scriptName,
 	}
@@ -294,7 +282,7 @@ func (t Transport) buildEnv(r *http.Request) (map[string]string, error) {
 	// PATH_TRANSLATED should only exist if PATH_INFO is defined.
 	// Info: https://www.ietf.org/rfc/rfc3875 Page 14
 	if env["PATH_INFO"] != "" {
-		env["PATH_TRANSLATED"] = filepath.Join(root, pathInfo) // Info: http://www.oreilly.com/openbook/cgi/ch02_04.html
+		env["PATH_TRANSLATED"] = caddyhttp.SanitizedPathJoin(root, pathInfo) // Info: http://www.oreilly.com/openbook/cgi/ch02_04.html
 	}
 
 	// compliance with the CGI specification requires that

modules/caddyhttp/reverseproxy/healthchecks.go

@@ -18,7 +18,6 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
 	"net"
 	"net/http"
@@ -189,13 +188,14 @@ func (h *Handler) doActiveHealthCheckForAllHosts() {
 				return
 			}
 			hostAddr := addr.JoinHostPort(0)
+			dialAddr := hostAddr
 			if addr.IsUnixNetwork() {
 				// this will be used as the Host portion of a http.Request URL, and
 				// paths to socket files would produce an error when creating URL,
 				// so use a fake Host value instead; unix sockets are usually local
 				hostAddr = "localhost"
 			}
-			err = h.doActiveHealthCheck(DialInfo{Network: addr.Network, Address: hostAddr}, hostAddr, upstream.Host)
+			err = h.doActiveHealthCheck(DialInfo{Network: addr.Network, Address: dialAddr}, hostAddr, upstream.Host)
 			if err != nil {
 				h.HealthChecks.Active.logger.Error("active health check failed",
 					zap.String("address", hostAddr),
@@ -281,7 +281,7 @@ func (h *Handler) doActiveHealthCheck(dialInfo DialInfo, hostAddr string, host H
 	}
 	defer func() {
 		// drain any remaining body so connection could be re-used
-		_, _ = io.Copy(ioutil.Discard, body)
+		_, _ = io.Copy(io.Discard, body)
 		resp.Body.Close()
 	}()
 
@@ -312,7 +312,7 @@ func (h *Handler) doActiveHealthCheck(dialInfo DialInfo, hostAddr string, host H
 
 	// if body does not match regex, mark down
 	if h.HealthChecks.Active.bodyRegexp != nil {
-		bodyBytes, err := ioutil.ReadAll(body)
+		bodyBytes, err := io.ReadAll(body)
 		if err != nil {
 			h.HealthChecks.Active.logger.Info("failed to read response body",
 				zap.String("host", hostAddr),

modules/caddyhttp/reverseproxy/httptransport.go

@@ -20,10 +20,10 @@ import (
 	"crypto/x509"
 	"encoding/base64"
 	"fmt"
-	"io/ioutil"
 	weakrand "math/rand"
 	"net"
 	"net/http"
+	"os"
 	"reflect"
 	"time"
 
@@ -62,9 +62,6 @@ type HTTPTransport struct {
 	// Maximum number of connections per host. Default: 0 (no limit)
 	MaxConnsPerHost int `json:"max_conns_per_host,omitempty"`
 
-	// Maximum number of idle connections per host. Default: 0 (uses Go's default of 2)
-	MaxIdleConnsPerHost int `json:"max_idle_conns_per_host,omitempty"`
-
 	// How long to wait before timing out trying to connect to
 	// an upstream.
 	DialTimeout caddy.Duration `json:"dial_timeout,omitempty"`
@@ -197,7 +194,6 @@ func (h *HTTPTransport) NewTransport(ctx caddy.Context) (*http.Transport, error)
 			return conn, nil
 		},
 		MaxConnsPerHost:        h.MaxConnsPerHost,
-		MaxIdleConnsPerHost:    h.MaxIdleConnsPerHost,
 		ResponseHeaderTimeout:  time.Duration(h.ResponseHeaderTimeout),
 		ExpectContinueTimeout:  time.Duration(h.ExpectContinueTimeout),
 		MaxResponseHeaderBytes: h.MaxResponseHeaderSize,
@@ -368,7 +364,7 @@ func (t TLSConfig) MakeTLSClientConfig(ctx caddy.Context) (*tls.Config, error) {
 			rootPool.AddCert(caCert)
 		}
 		for _, pemFile := range t.RootCAPEMFiles {
-			pemData, err := ioutil.ReadFile(pemFile)
+			pemData, err := os.ReadFile(pemFile)
 			if err != nil {
 				return nil, fmt.Errorf("failed reading ca cert: %v", err)
 			}
@@ -412,13 +408,13 @@ type KeepAlive struct {
 	// How often to probe for liveness.
 	ProbeInterval caddy.Duration `json:"probe_interval,omitempty"`
 
-	// Maximum number of idle connections.
+	// Maximum number of idle connections. Default: 0, which means no limit.
 	MaxIdleConns int `json:"max_idle_conns,omitempty"`
 
-	// Maximum number of idle connections per upstream host.
+	// Maximum number of idle connections per host. Default: 32.
 	MaxIdleConnsPerHost int `json:"max_idle_conns_per_host,omitempty"`
 
-	// How long connections should be kept alive when idle.
+	// How long connections should be kept alive when idle. Default: 0, which means no timeout.
 	IdleConnTimeout caddy.Duration `json:"idle_timeout,omitempty"`
 }
 

modules/caddyhttp/reverseproxy/reverseproxy.go

@@ -23,6 +23,7 @@ import (
 	"io"
 	"net"
 	"net/http"
+	"net/textproto"
 	"net/url"
 	"regexp"
 	"strconv"
@@ -80,10 +81,13 @@ type Handler struct {
 	// Upstreams is the list of backends to proxy to.
 	Upstreams UpstreamPool `json:"upstreams,omitempty"`
 
-	// Adjusts how often to flush the response buffer. A
-	// negative value disables response buffering.
-	// TODO: figure out good defaults and write docs for this
-	// (see https://github.com/caddyserver/caddy/issues/1460)
+	// Adjusts how often to flush the response buffer. By default,
+	// no periodic flushing is done. A negative value disables
+	// response buffering, and flushes immediately after each
+	// write to the client. This option is ignored when the upstream's
+	// response is recognized as a streaming response, or if its
+	// content length is -1; for such responses, writes are flushed
+	// to the client immediately.
 	FlushInterval caddy.Duration `json:"flush_interval,omitempty"`
 
 	// Headers manipulates headers between Caddy and the backend.
@@ -204,7 +208,7 @@ func (h *Handler) Provision(ctx caddy.Context) error {
 			KeepAlive: &KeepAlive{
 				ProbeInterval:       caddy.Duration(30 * time.Second),
 				IdleConnTimeout:     caddy.Duration(2 * time.Minute),
-				MaxIdleConnsPerHost: 32,
+				MaxIdleConnsPerHost: 32, // seems about optimal, see #2805
 			},
 			DialTimeout: caddy.Duration(10 * time.Second),
 		}
@@ -391,9 +395,23 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyht
 	// should not permanently change r.Host; issue #3509)
 	reqHost := r.Host
 	reqHeader := r.Header
+
+	// sanitize the request URL; we expect it to not contain the scheme and host
+	// since those should be determined by r.TLS and r.Host respectively, but
+	// some clients may include it in the request-line, which is technically
+	// valid in HTTP, but breaks reverseproxy behaviour, overriding how the
+	// dialer will behave. See #4237 for context.
+	origURLScheme := r.URL.Scheme
+	origURLHost := r.URL.Host
+	r.URL.Scheme = ""
+	r.URL.Host = ""
+
+	// restore modifications to the request after we're done proxying
 	defer func() {
 		r.Host = reqHost     // TODO: data race, see #4038
 		r.Header = reqHeader // TODO: data race, see #4038
+		r.URL.Scheme = origURLScheme
+		r.URL.Host = origURLHost
 	}()
 
 	start := time.Now()
@@ -504,18 +522,14 @@ func (h Handler) prepareRequest(req *http.Request) error {
 	// Remove hop-by-hop headers to the backend. Especially
 	// important is "Connection" because we want a persistent
 	// connection, regardless of what the client sent to us.
+	// Issue golang/go#46313: don't skip if field is empty.
 	for _, h := range hopHeaders {
-		hv := req.Header.Get(h)
-		if hv == "" {
-			continue
-		}
-		if h == "Te" && hv == "trailers" {
-			// Issue golang/go#21096: tell backend applications that
-			// care about trailer support that we support
-			// trailers. (We do, but we don't go out of
-			// our way to advertise that unless the
-			// incoming client request thought it was
-			// worth mentioning)
+		// Issue golang/go#21096: tell backend applications that care about trailer support
+		// that we support trailers. (We do, but we don't go out of our way to
+		// advertise that unless the incoming client request thought it was worth
+		// mentioning.)
+		if h == "Te" && httpguts.HeaderValuesContainsToken(req.Header["Te"], "trailers") {
+			req.Header.Set("Te", "trailers")
 			continue
 		}
 		req.Header.Del(h)
@@ -532,13 +546,19 @@ func (h Handler) prepareRequest(req *http.Request) error {
 		// If we aren't the first proxy retain prior
 		// X-Forwarded-For information as a comma+space
 		// separated list and fold multiple headers into one.
-		if prior, ok := req.Header["X-Forwarded-For"]; ok {
+		prior, ok := req.Header["X-Forwarded-For"]
+		omit := ok && prior == nil // Issue 38079: nil now means don't populate the header
+		if len(prior) > 0 {
 			clientIP = strings.Join(prior, ", ") + ", " + clientIP
 		}
-		req.Header.Set("X-Forwarded-For", clientIP)
+		if !omit {
+			req.Header.Set("X-Forwarded-For", clientIP)
+		}
 	}
 
-	if req.Header.Get("X-Forwarded-Proto") == "" {
+	prior, ok := req.Header["X-Forwarded-Proto"]
+	omit := ok && prior == nil
+	if len(prior) == 0 && !omit {
 		// set X-Forwarded-Proto; many backend apps expect this too
 		proto := "https"
 		if req.TLS == nil {
@@ -568,12 +588,11 @@ func (h *Handler) reverseProxy(rw http.ResponseWriter, req *http.Request, repl *
 	duration := time.Since(start)
 	logger := h.logger.With(
 		zap.String("upstream", di.Upstream.String()),
+		zap.Duration("duration", duration),
 		zap.Object("request", caddyhttp.LoggableHTTPRequest{Request: req}),
 	)
 	if err != nil {
-		logger.Debug("upstream roundtrip",
-			zap.Duration("duration", duration),
-			zap.Error(err))
+		logger.Debug("upstream roundtrip", zap.Error(err))
 		return err
 	}
 	logger.Debug("upstream roundtrip",
@@ -609,6 +628,11 @@ func (h *Handler) reverseProxy(rw http.ResponseWriter, req *http.Request, repl *
 		res.Body = h.bufferedBody(res.Body)
 	}
 
+	// the response body may get closed by a response handler,
+	// and we need to keep track to make sure we don't try to copy
+	// the response if it was already closed
+	bodyClosed := false
+
 	// see if any response handler is configured for this response from the backend
 	for i, rh := range h.HandleResponse {
 		if rh.Match != nil && !rh.Match.Match(res.StatusCode, res.Header) {
@@ -633,8 +657,6 @@ func (h *Handler) reverseProxy(rw http.ResponseWriter, req *http.Request, repl *
 			continue
 		}
 
-		res.Body.Close()
-
 		// set up the replacer so that parts of the original response can be
 		// used for routing decisions
 		for field, value := range res.Header {
@@ -644,7 +666,17 @@ func (h *Handler) reverseProxy(rw http.ResponseWriter, req *http.Request, repl *
 		repl.Set("http.reverse_proxy.status_text", res.Status)
 
 		h.logger.Debug("handling response", zap.Int("handler", i))
-		if routeErr := rh.Routes.Compile(next).ServeHTTP(rw, req); routeErr != nil {
+
+		// pass the request through the response handler routes
+		routeErr := rh.Routes.Compile(next).ServeHTTP(rw, req)
+
+		// always close the response body afterwards since it's expected
+		// that the response handler routes will have written to the
+		// response writer with a new body
+		res.Body.Close()
+		bodyClosed = true
+
+		if routeErr != nil {
 			// wrap error in roundtripSucceeded so caller knows that
 			// the roundtrip was successful and to not retry
 			return roundtripSucceeded{routeErr}
@@ -685,25 +717,18 @@ func (h *Handler) reverseProxy(rw http.ResponseWriter, req *http.Request, repl *
 	}
 
 	rw.WriteHeader(res.StatusCode)
-
-	// some apps need the response headers before starting to stream content with http2,
-	// so it's important to explicitly flush the headers to the client before streaming the data.
-	// (see https://github.com/caddyserver/caddy/issues/3556 for use case and nuances)
-	if h.isBidirectionalStream(req, res) {
-		if wf, ok := rw.(http.Flusher); ok {
-			wf.Flush()
+	if !bodyClosed {
+		err = h.copyResponse(rw, res.Body, h.flushInterval(req, res))
+		res.Body.Close() // close now, instead of defer, to populate res.Trailer
+		if err != nil {
+			// we're streaming the response and we've already written headers, so
+			// there's nothing an error handler can do to recover at this point;
+			// the standard lib's proxy panics at this point, but we'll just log
+			// the error and abort the stream here
+			h.logger.Error("aborting with incomplete response", zap.Error(err))
+			return nil
 		}
 	}
-	err = h.copyResponse(rw, res.Body, h.flushInterval(req, res))
-	res.Body.Close() // close now, instead of defer, to populate res.Trailer
-	if err != nil {
-		// we're streaming the response and we've already written headers, so
-		// there's nothing an error handler can do to recover at this point;
-		// the standard lib's proxy panics at this point, but we'll just log
-		// the error and abort the stream here
-		h.logger.Error("aborting with incomplete response", zap.Error(err))
-		return nil
-	}
 
 	if len(res.Trailer) > 0 {
 		// Force chunking if we saw a response trailer.
@@ -831,10 +856,10 @@ func upgradeType(h http.Header) string {
 // removeConnectionHeaders removes hop-by-hop headers listed in the "Connection" header of h.
 // See RFC 7230, section 6.1
 func removeConnectionHeaders(h http.Header) {
-	if c := h.Get("Connection"); c != "" {
-		for _, f := range strings.Split(c, ",") {
-			if f = strings.TrimSpace(f); f != "" {
-				h.Del(f)
+	for _, f := range h["Connection"] {
+		for _, sf := range strings.Split(f, ",") {
+			if sf = textproto.TrimString(sf); sf != "" {
+				h.Del(sf)
 			}
 		}
 	}

modules/caddyhttp/reverseproxy/streaming.go

@@ -32,6 +32,9 @@ import (
 func (h Handler) handleUpgradeResponse(logger *zap.Logger, rw http.ResponseWriter, req *http.Request, res *http.Response) {
 	reqUpType := upgradeType(req.Header)
 	resUpType := upgradeType(res.Header)
+	// TODO: Update to use "net/http/internal/ascii" once we bumped
+	// the minimum Go version to 1.17.
+	// See https://github.com/golang/go/commit/5c489514bc5e61ad9b5b07bd7d8ec65d66a0512a
 	if reqUpType != resUpType {
 		h.logger.Debug("backend tried to switch to unexpected protocol via Upgrade header",
 			zap.String("backend_upgrade", resUpType),
@@ -39,8 +42,6 @@ func (h Handler) handleUpgradeResponse(logger *zap.Logger, rw http.ResponseWrite
 		return
 	}
 
-	copyHeader(res.Header, rw.Header())
-
 	hj, ok := rw.(http.Hijacker)
 	if !ok {
 		h.logger.Sugar().Errorf("can't switch protocols using non-Hijacker ResponseWriter type %T", rw)
@@ -78,6 +79,9 @@ func (h Handler) handleUpgradeResponse(logger *zap.Logger, rw http.ResponseWrite
 		logger.Debug("connection closed", zap.Duration("duration", time.Since(start)))
 	}()
 
+	copyHeader(rw.Header(), res.Header)
+
+	res.Header = rw.Header()
 	res.Body = nil // so res.Write only writes the headers; we have res.Body in backConn above
 	if err := res.Write(brw); err != nil {
 		h.logger.Debug("response write", zap.Error(err))
@@ -107,13 +111,16 @@ func (h Handler) flushInterval(req *http.Request, res *http.Response) time.Durat
 		return -1 // negative means immediately
 	}
 
+	// We might have the case of streaming for which Content-Length might be unset.
+	if res.ContentLength == -1 {
+		return -1
+	}
+
 	// for h2 and h2c upstream streaming data to client (issues #3556 and #3606)
 	if h.isBidirectionalStream(req, res) {
 		return -1
 	}
 
-	// TODO: more specific cases? e.g. res.ContentLength == -1? (this TODO is from the std lib, but
-	// strangely similar to our isBidirectionalStream function that we implemented ourselves)
 	return time.Duration(h.FlushInterval)
 }
 
@@ -142,6 +149,11 @@ func (h Handler) copyResponse(dst io.Writer, src io.Reader, flushInterval time.D
 				latency: flushInterval,
 			}
 			defer mlw.stop()
+
+			// set up initial timer so headers get flushed even if body writes are delayed
+			mlw.flushPending = true
+			mlw.t = time.AfterFunc(flushInterval, mlw.delayedFlush)
+
 			dst = mlw
 		}
 	}

modules/caddyhttp/routes.go

@@ -200,6 +200,15 @@ func wrapRoute(route Route) Middleware {
 
 			// route must match at least one of the matcher sets
 			if !route.MatcherSets.AnyMatch(req) {
+				// allow matchers the opportunity to short circuit
+				// the request and trigger the error handling chain
+				err, ok := GetVar(req.Context(), MatcherErrorVarKey).(error)
+				if ok {
+					return err
+				}
+
+				// call the next handler, and skip this one,
+				// since the matcher didn't match
 				return nextCopy.ServeHTTP(rw, req)
 			}
 

modules/caddyhttp/server.go

@@ -184,8 +184,11 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 				log = logger.Error
 			}
 
+			userID, _ := repl.GetString("http.auth.user.id")
+
 			log("handled request",
 				zap.String("common_log", repl.ReplaceAll(commonLogFormat, commonLogEmptyValue)),
+				zap.String("user_id", userID),
 				zap.Duration("duration", duration),
 				zap.Int("size", wrec.Size()),
 				zap.Int("status", wrec.Status()),
@@ -379,7 +382,9 @@ func (s *Server) hasTLSClientAuth() bool {
 // that it is after any other host matcher but before any "catch-all"
 // route without a host matcher.
 func (s *Server) findLastRouteWithHostMatcher() int {
+	foundHostMatcher := false
 	lastIndex := len(s.Routes)
+
 	for i, route := range s.Routes {
 		// since we want to break out of an inner loop, use a closure
 		// to allow us to use 'return' when we found a host matcher
@@ -388,6 +393,7 @@ func (s *Server) findLastRouteWithHostMatcher() int {
 				for _, matcher := range sets {
 					switch matcher.(type) {
 					case *MatchHost:
+						foundHostMatcher = true
 						return true
 					}
 				}
@@ -401,6 +407,14 @@ func (s *Server) findLastRouteWithHostMatcher() int {
 			lastIndex = i + 1
 		}
 	}
+
+	// If we didn't actually find a host matcher, return 0
+	// because that means every defined route was a "catch-all".
+	// See https://caddy.community/t/how-to-set-priority-in-caddyfile/13002/8
+	if !foundHostMatcher {
+		return 0
+	}
+
 	return lastIndex
 }
 

modules/caddyhttp/staticresp_test.go

@@ -16,7 +16,7 @@ package caddyhttp
 
 import (
 	"context"
-	"io/ioutil"
+	"io"
 	"net/http"
 	"net/http/httptest"
 	"strconv"
@@ -44,7 +44,7 @@ func TestStaticResponseHandler(t *testing.T) {
 	}
 
 	resp := w.Result()
-	respBody, _ := ioutil.ReadAll(resp.Body)
+	respBody, _ := io.ReadAll(resp.Body)
 
 	if resp.StatusCode != http.StatusNotFound {
 		t.Errorf("expected status %d but got %d", http.StatusNotFound, resp.StatusCode)

modules/caddyhttp/templates/templates.go

@@ -16,6 +16,7 @@ package templates
 
 import (
 	"bytes"
+	"errors"
 	"fmt"
 	"net/http"
 	"strconv"
@@ -90,9 +91,25 @@ func init() {
 // {{httpInclude "/foo/bar?q=val"}}
 // ```
 //
+// ##### `import`
+//
+// Imports the contents of another file and adds any template definitions to the template stack. If there are no defitions, the filepath will be the defition name. Any {{ define }} blocks will be accessible by {{ template }} or {{ block }}. Imports must happen before the template or block action is called
+//
+// **filename.html**
+// ```
+// {{ define "main" }}
+// content
+// {{ end }}
+//
+// **index.html**
+// ```
+// {{ import "/path/to/file.html" }}
+// {{ template "main" }}
+// ```
+//
 // ##### `include`
 //
-// Includes the contents of another file. Optionally can pass key-value pairs as arguments to be accessed by the included file.
+// Includes the contents of another file and renders in-place. Optionally can pass key-value pairs as arguments to be accessed by the included file.
 //
 // ```
 // {{include "path/to/file.html"}}  // no arguments
@@ -220,7 +237,8 @@ type Templates struct {
 	// Default is text/plain, text/markdown, and text/html.
 	MIMETypes []string `json:"mime_types,omitempty"`
 
-	// The template action delimiters.
+	// The template action delimiters. If set, must be precisely two elements:
+	// the opening and closing delimiters. Default: `["{{", "}}"]`
 	Delimiters []string `json:"delimiters,omitempty"`
 }
 
@@ -312,6 +330,12 @@ func (t *Templates) executeTemplate(rr caddyhttp.ResponseRecorder, r *http.Reque
 
 	err := ctx.executeTemplateInBuffer(r.URL.Path, rr.Buffer())
 	if err != nil {
+		// templates may return a custom HTTP error to be propagated to the client,
+		// otherwise for any other error we assume the template is broken
+		var handlerErr caddyhttp.HandlerError
+		if errors.As(err, &handlerErr) {
+			return handlerErr
+		}
 		return caddyhttp.Error(http.StatusInternalServerError, err)
 	}
 

modules/caddyhttp/templates/tplcontext.go

@@ -46,24 +46,26 @@ type TemplateContext struct {
 	RespHeader WrappedHeader
 
 	config *Templates
+	tpl    *template.Template
 }
 
 // NewTemplate returns a new template intended to be evaluated with this
 // context, as it is initialized with configuration from this context.
-func (c TemplateContext) NewTemplate(tplName string) *template.Template {
-	tpl := template.New(tplName)
+func (c *TemplateContext) NewTemplate(tplName string) *template.Template {
+	c.tpl = template.New(tplName)
 
 	// customize delimiters, if applicable
 	if c.config != nil && len(c.config.Delimiters) == 2 {
-		tpl.Delims(c.config.Delimiters[0], c.config.Delimiters[1])
+		c.tpl.Delims(c.config.Delimiters[0], c.config.Delimiters[1])
 	}
 
 	// add sprig library
-	tpl.Funcs(sprigFuncMap)
+	c.tpl.Funcs(sprigFuncMap)
 
 	// add our own library
-	tpl.Funcs(template.FuncMap{
+	c.tpl.Funcs(template.FuncMap{
 		"include":          c.funcInclude,
+		"import":           c.funcImport,
 		"httpInclude":      c.funcHTTPInclude,
 		"stripHTML":        c.funcStripHTML,
 		"markdown":         c.funcMarkdown,
@@ -74,8 +76,7 @@ func (c TemplateContext) NewTemplate(tplName string) *template.Template {
 		"fileExists":       c.funcFileExists,
 		"httpError":        c.funcHTTPError,
 	})
-
-	return tpl
+	return c.tpl
 }
 
 // OriginalReq returns the original, unmodified, un-rewritten request as
@@ -85,26 +86,13 @@ func (c TemplateContext) OriginalReq() http.Request {
 	return or
 }
 
-// funcInclude returns the contents of filename relative to the site root.
+// funcInclude returns the contents of filename relative to the site root and renders it in place.
 // Note that included files are NOT escaped, so you should only include
 // trusted files. If it is not trusted, be sure to use escaping functions
 // in your template.
 func (c TemplateContext) funcInclude(filename string, args ...interface{}) (string, error) {
-	if c.Root == nil {
-		return "", fmt.Errorf("root file system not specified")
-	}
+	bodyBuf, err := c.readFileToBuffer(filename)
 
-	file, err := c.Root.Open(filename)
-	if err != nil {
-		return "", err
-	}
-	defer file.Close()
-
-	bodyBuf := bufPool.Get().(*bytes.Buffer)
-	bodyBuf.Reset()
-	defer bufPool.Put(bodyBuf)
-
-	_, err = io.Copy(bodyBuf, file)
 	if err != nil {
 		return "", err
 	}
@@ -119,6 +107,30 @@ func (c TemplateContext) funcInclude(filename string, args ...interface{}) (stri
 	return bodyBuf.String(), nil
 }
 
+// readFileToBuffer returns the contents of filename relative to root as a buffer
+func (c TemplateContext) readFileToBuffer(filename string) (*bytes.Buffer, error) {
+	if c.Root == nil {
+		return nil, fmt.Errorf("root file system not specified")
+	}
+
+	file, err := c.Root.Open(filename)
+	if err != nil {
+		return nil, err
+	}
+	defer file.Close()
+
+	bodyBuf := bufPool.Get().(*bytes.Buffer)
+	bodyBuf.Reset()
+	defer bufPool.Put(bodyBuf)
+
+	_, err = io.Copy(bodyBuf, file)
+	if err != nil {
+		return nil, err
+	}
+
+	return bodyBuf, nil
+}
+
 // funcHTTPInclude returns the body of a virtual (lightweight) request
 // to the given URI on the same server. Note that included bodies
 // are NOT escaped, so you should only include trusted resources.
@@ -148,6 +160,7 @@ func (c TemplateContext) funcHTTPInclude(uri string) (string, error) {
 	}
 	virtReq.Host = c.Req.Host
 	virtReq.Header = c.Req.Header.Clone()
+	virtReq.Header.Set("Accept-Encoding", "identity") // https://github.com/caddyserver/caddy/issues/4352
 	virtReq.Trailer = c.Req.Trailer.Clone()
 	virtReq.Header.Set(recursionPreventionHeader, strconv.Itoa(recursionCount))
 
@@ -167,17 +180,34 @@ func (c TemplateContext) funcHTTPInclude(uri string) (string, error) {
 	return buf.String(), nil
 }
 
-func (c TemplateContext) executeTemplateInBuffer(tplName string, buf *bytes.Buffer) error {
-	tpl := c.NewTemplate(tplName)
+// funcImport parses the filename into the current template stack. The imported
+// file will be rendered within the current template by calling {{ block }} or
+// {{ template }} from the standard template library. If the imported file has
+// no {{ define }} blocks, the name of the import will be the path
+func (c *TemplateContext) funcImport(filename string) (string, error) {
+	bodyBuf, err := c.readFileToBuffer(filename)
+	if err != nil {
+		return "", err
+	}
 
-	parsedTpl, err := tpl.Parse(buf.String())
+	_, err = c.tpl.Parse(bodyBuf.String())
+	if err != nil {
+		return "", err
+	}
+	return "", nil
+}
+
+func (c *TemplateContext) executeTemplateInBuffer(tplName string, buf *bytes.Buffer) error {
+	c.NewTemplate(tplName)
+
+	_, err := c.tpl.Parse(buf.String())
 	if err != nil {
 		return err
 	}
 
 	buf.Reset() // reuse buffer for output
 
-	return parsedTpl.Execute(buf, c)
+	return c.tpl.Execute(buf, c)
 }
 
 func (c TemplateContext) funcPlaceholder(name string) string {
@@ -350,9 +380,8 @@ func (c TemplateContext) funcFileExists(filename string) (bool, error) {
 	return false, nil
 }
 
-// funcHTTPError returns a structured HTTP handler error. EXPERIMENTAL.
-// TODO: Requires https://github.com/golang/go/issues/34201 to be fixed (Go 1.17).
-// Example usage might be: `{{if not (fileExists $includeFile)}}{{httpError 404}}{{end}}`
+// funcHTTPError returns a structured HTTP handler error. EXPERIMENTAL; SUBJECT TO CHANGE.
+// Example usage: `{{if not (fileExists $includeFile)}}{{httpError 404}}{{end}}`
 func (c TemplateContext) funcHTTPError(statusCode int) (bool, error) {
 	return false, caddyhttp.Error(statusCode, nil)
 }

modules/caddyhttp/templates/tplcontext_test.go

@@ -16,8 +16,8 @@ package templates
 
 import (
 	"bytes"
+	"context"
 	"fmt"
-	"io/ioutil"
 	"net/http"
 	"os"
 	"path/filepath"
@@ -26,10 +26,49 @@ import (
 	"strings"
 	"testing"
 	"time"
+
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 )
 
+type handle struct {
+}
+
+func (h *handle) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	if r.Header.Get("Accept-Encoding") == "identity" {
+		w.Write([]byte("good contents"))
+	} else {
+		w.Write([]byte("bad cause Accept-Encoding: " + r.Header.Get("Accept-Encoding")))
+	}
+}
+
+func TestHTTPInclude(t *testing.T) {
+	tplContext := getContextOrFail(t)
+	for i, test := range []struct {
+		uri     string
+		handler *handle
+		expect  string
+	}{
+		{
+			uri:     "https://example.com/foo/bar",
+			handler: &handle{},
+			expect:  "good contents",
+		},
+	} {
+		ctx := context.WithValue(tplContext.Req.Context(), caddyhttp.ServerCtxKey, test.handler)
+		tplContext.Req = tplContext.Req.WithContext(ctx)
+		tplContext.Req.Header.Add("Accept-Encoding", "gzip")
+		result, err := tplContext.funcHTTPInclude(test.uri)
+		if result != test.expect {
+			t.Errorf("Test %d: expected '%s' but got '%s'", i, test.expect, result)
+		}
+		if err != nil {
+			t.Errorf("Test %d: got error: %v", i, result)
+		}
+	}
+}
+
 func TestMarkdown(t *testing.T) {
-	context := getContextOrFail(t)
+	tplContext := getContextOrFail(t)
 
 	for i, test := range []struct {
 		body   string
@@ -40,7 +79,7 @@ func TestMarkdown(t *testing.T) {
 			expect: "<ul>\n<li>str1</li>\n<li>str2</li>\n</ul>\n",
 		},
 	} {
-		result, err := context.funcMarkdown(test.body)
+		result, err := tplContext.funcMarkdown(test.body)
 		if result != test.expect {
 			t.Errorf("Test %d: expected '%s' but got '%s'", i, test.expect, result)
 		}
@@ -81,9 +120,9 @@ func TestCookie(t *testing.T) {
 			expect:     "cookieValue",
 		},
 	} {
-		context := getContextOrFail(t)
-		context.Req.AddCookie(test.cookie)
-		actual := context.Cookie(test.cookieName)
+		tplContext := getContextOrFail(t)
+		tplContext.Req.AddCookie(test.cookie)
+		actual := tplContext.Cookie(test.cookieName)
 		if actual != test.expect {
 			t.Errorf("Test %d: Expected cookie value '%s' but got '%s' for cookie with name '%s'",
 				i, test.expect, actual, test.cookieName)
@@ -91,13 +130,147 @@ func TestCookie(t *testing.T) {
 	}
 }
 
+func TestImport(t *testing.T) {
+	for i, test := range []struct {
+		fileContent string
+		fileName    string
+		shouldErr   bool
+		expect      string
+	}{
+		{
+			// file exists, template is defined
+			fileContent: `{{ define "imported" }}text{{end}}`,
+			fileName:    "file1",
+			shouldErr:   false,
+			expect:      `"imported"`,
+		},
+		{
+			// file does not exit
+			fileContent: "",
+			fileName:    "",
+			shouldErr:   true,
+		},
+	} {
+		tplContext := getContextOrFail(t)
+		var absFilePath string
+
+		// create files for test case
+		if test.fileName != "" {
+			absFilePath := filepath.Join(fmt.Sprintf("%s", tplContext.Root), test.fileName)
+			if err := os.WriteFile(absFilePath, []byte(test.fileContent), os.ModePerm); err != nil {
+				os.Remove(absFilePath)
+				t.Fatalf("Test %d: Expected no error creating file, got: '%s'", i, err.Error())
+			}
+		}
+
+		// perform test
+		tplContext.NewTemplate("parent")
+		actual, err := tplContext.funcImport(test.fileName)
+		templateWasDefined := strings.Contains(tplContext.tpl.DefinedTemplates(), test.expect)
+		if err != nil {
+			if !test.shouldErr {
+				t.Errorf("Test %d: Expected no error, got: '%s'", i, err)
+			}
+		} else if test.shouldErr {
+			t.Errorf("Test %d: Expected error but had none", i)
+		} else if !templateWasDefined && actual != "" {
+			// template should be defined, return value should be an empty string
+			t.Errorf("Test %d: Expected template %s to be define but got %s", i, test.expect, tplContext.tpl.DefinedTemplates())
+
+		}
+
+		if absFilePath != "" {
+			if err := os.Remove(absFilePath); err != nil && !os.IsNotExist(err) {
+				t.Fatalf("Test %d: Expected no error removing temporary test file, got: %v", i, err)
+			}
+		}
+	}
+}
+
+func TestInclude(t *testing.T) {
+	for i, test := range []struct {
+		fileContent string
+		fileName    string
+		shouldErr   bool
+		expect      string
+		args        string
+	}{
+		{
+			// file exists, content is text only
+			fileContent: "text",
+			fileName:    "file1",
+			shouldErr:   false,
+			expect:      "text",
+		},
+		{
+			// file exists, content is template
+			fileContent: "{{ if . }}text{{ end }}",
+			fileName:    "file1",
+			shouldErr:   false,
+			expect:      "text",
+		},
+		{
+			// file does not exit
+			fileContent: "",
+			fileName:    "",
+			shouldErr:   true,
+		},
+		{
+			// args
+			fileContent: "{{ index .Args 0 }}",
+			fileName:    "file1",
+			shouldErr:   false,
+			args:        "text",
+			expect:      "text",
+		},
+		{
+			// args, reference arg out of range
+			fileContent: "{{ index .Args 1 }}",
+			fileName:    "file1",
+			shouldErr:   true,
+			args:        "text",
+		},
+	} {
+		tplContext := getContextOrFail(t)
+		var absFilePath string
+
+		// create files for test case
+		if test.fileName != "" {
+			absFilePath := filepath.Join(fmt.Sprintf("%s", tplContext.Root), test.fileName)
+			if err := os.WriteFile(absFilePath, []byte(test.fileContent), os.ModePerm); err != nil {
+				os.Remove(absFilePath)
+				t.Fatalf("Test %d: Expected no error creating file, got: '%s'", i, err.Error())
+			}
+		}
+
+		// perform test
+		actual, err := tplContext.funcInclude(test.fileName, test.args)
+		if err != nil {
+			if !test.shouldErr {
+				t.Errorf("Test %d: Expected no error, got: '%s'", i, err)
+			}
+		} else if test.shouldErr {
+			t.Errorf("Test %d: Expected error but had none", i)
+		} else if actual != test.expect {
+			t.Errorf("Test %d: Expected %s but got %s", i, test.expect, actual)
+
+		}
+
+		if absFilePath != "" {
+			if err := os.Remove(absFilePath); err != nil && !os.IsNotExist(err) {
+				t.Fatalf("Test %d: Expected no error removing temporary test file, got: %v", i, err)
+			}
+		}
+	}
+}
+
 func TestCookieMultipleCookies(t *testing.T) {
-	context := getContextOrFail(t)
+	tplContext := getContextOrFail(t)
 
 	cookieNameBase, cookieValueBase := "cookieName", "cookieValue"
 
 	for i := 0; i < 10; i++ {
-		context.Req.AddCookie(&http.Cookie{
+		tplContext.Req.AddCookie(&http.Cookie{
 			Name:  fmt.Sprintf("%s%d", cookieNameBase, i),
 			Value: fmt.Sprintf("%s%d", cookieValueBase, i),
 		})
@@ -105,7 +278,7 @@ func TestCookieMultipleCookies(t *testing.T) {
 
 	for i := 0; i < 10; i++ {
 		expectedCookieVal := fmt.Sprintf("%s%d", cookieValueBase, i)
-		actualCookieVal := context.Cookie(fmt.Sprintf("%s%d", cookieNameBase, i))
+		actualCookieVal := tplContext.Cookie(fmt.Sprintf("%s%d", cookieNameBase, i))
 		if actualCookieVal != expectedCookieVal {
 			t.Errorf("Expected cookie value %s, found %s", expectedCookieVal, actualCookieVal)
 		}
@@ -113,7 +286,7 @@ func TestCookieMultipleCookies(t *testing.T) {
 }
 
 func TestIP(t *testing.T) {
-	context := getContextOrFail(t)
+	tplContext := getContextOrFail(t)
 	for i, test := range []struct {
 		inputRemoteAddr string
 		expect          string
@@ -124,15 +297,15 @@ func TestIP(t *testing.T) {
 		{"[2001:db8:a0b:12f0::1]", "[2001:db8:a0b:12f0::1]"},
 		{`[fe80:1::3%eth0]:44`, `fe80:1::3%eth0`},
 	} {
-		context.Req.RemoteAddr = test.inputRemoteAddr
-		if actual := context.RemoteIP(); actual != test.expect {
+		tplContext.Req.RemoteAddr = test.inputRemoteAddr
+		if actual := tplContext.RemoteIP(); actual != test.expect {
 			t.Errorf("Test %d: Expected %s but got %s", i, test.expect, actual)
 		}
 	}
 }
 
 func TestStripHTML(t *testing.T) {
-	context := getContextOrFail(t)
+	tplContext := getContextOrFail(t)
 
 	for i, test := range []struct {
 		input  string
@@ -169,7 +342,7 @@ func TestStripHTML(t *testing.T) {
 			expect: `<h1hi`,
 		},
 	} {
-		actual := context.funcStripHTML(test.input)
+		actual := tplContext.funcStripHTML(test.input)
 		if actual != test.expect {
 			t.Errorf("Test %d: Expected %s, found %s. Input was StripHTML(%s)", i, test.expect, actual, test.input)
 		}
@@ -217,19 +390,19 @@ func TestFileListing(t *testing.T) {
 			verifyErr: os.IsNotExist,
 		},
 	} {
-		context := getContextOrFail(t)
+		tplContext := getContextOrFail(t)
 		var dirPath string
 		var err error
 
 		// create files for test case
 		if test.fileNames != nil {
-			dirPath, err = ioutil.TempDir(fmt.Sprintf("%s", context.Root), "caddy_ctxtest")
+			dirPath, err = os.MkdirTemp(fmt.Sprintf("%s", tplContext.Root), "caddy_ctxtest")
 			if err != nil {
 				t.Fatalf("Test %d: Expected no error creating directory, got: '%s'", i, err.Error())
 			}
 			for _, name := range test.fileNames {
 				absFilePath := filepath.Join(dirPath, name)
-				if err = ioutil.WriteFile(absFilePath, []byte(""), os.ModePerm); err != nil {
+				if err = os.WriteFile(absFilePath, []byte(""), os.ModePerm); err != nil {
 					os.RemoveAll(dirPath)
 					t.Fatalf("Test %d: Expected no error creating file, got: '%s'", i, err.Error())
 				}
@@ -238,7 +411,7 @@ func TestFileListing(t *testing.T) {
 
 		// perform test
 		input := filepath.ToSlash(filepath.Join(filepath.Base(dirPath), test.inputBase))
-		actual, err := context.funcListFiles(input)
+		actual, err := tplContext.funcListFiles(input)
 		if err != nil {
 			if !test.shouldErr {
 				t.Errorf("Test %d: Expected no error, got: '%s'", i, err)
@@ -271,7 +444,7 @@ func TestFileListing(t *testing.T) {
 }
 
 func TestSplitFrontMatter(t *testing.T) {
-	context := getContextOrFail(t)
+	tplContext := getContextOrFail(t)
 
 	for i, test := range []struct {
 		input  string
@@ -332,7 +505,7 @@ title = "Welcome"
 			body:   "\n### Test",
 		},
 	} {
-		result, _ := context.funcSplitFrontMatter(test.input)
+		result, _ := tplContext.funcSplitFrontMatter(test.input)
 		if result.Meta["title"] != test.expect {
 			t.Errorf("Test %d: Expected %s, found %s. Input was SplitFrontMatter(%s)", i, test.expect, result.Meta["title"], test.input)
 		}
@@ -344,11 +517,11 @@ title = "Welcome"
 }
 
 func getContextOrFail(t *testing.T) TemplateContext {
-	context, err := initTestContext()
+	tplContext, err := initTestContext()
 	if err != nil {
 		t.Fatalf("failed to prepare test context: %v", err)
 	}
-	return context
+	return tplContext
 }
 
 func initTestContext() (TemplateContext, error) {

modules/caddyhttp/vars.go

@@ -29,9 +29,14 @@ func init() {
 	caddy.RegisterModule(MatchVarsRE{})
 }
 
-// VarsMiddleware is an HTTP middleware which sets variables
-// in the context, mainly for use by placeholders. The
-// placeholders have the form: `{http.vars.variable_name}`
+// VarsMiddleware is an HTTP middleware which sets variables to
+// have values that can be used in the HTTP request handler
+// chain. The primary way to access variables is with placeholders,
+// which have the form: `{http.vars.variable_name}`, or with
+// the `vars` and `vars_regexp` request matchers.
+//
+// The key is the variable name, and the value is the value of the
+// variable. Both the name and value may use or contain placeholders.
 type VarsMiddleware map[string]string
 
 // CaddyModule returns the Caddy module information.

modules/caddypki/acmeserver/acmeserver.go

@@ -29,6 +29,7 @@ import (
 	"github.com/go-chi/chi"
 	"github.com/smallstep/certificates/acme"
 	acmeAPI "github.com/smallstep/certificates/acme/api"
+	acmeNoSQL "github.com/smallstep/certificates/acme/db/nosql"
 	"github.com/smallstep/certificates/authority"
 	"github.com/smallstep/certificates/authority/provisioner"
 	"github.com/smallstep/certificates/db"
@@ -49,17 +50,16 @@ type Handler struct {
 
 	// The hostname or IP address by which ACME clients
 	// will access the server. This is used to populate
-	// the ACME directory endpoint. Default: localhost.
+	// the ACME directory endpoint. If not set, the Host
+	// header of the request will be used.
 	// COMPATIBILITY NOTE / TODO: This property may go away in the
-	// future, as it is currently only required due to
-	// limitations in the underlying library. Do not rely
-	// on this property long-term; check release notes.
+	// future. Do not rely on this property long-term; check release notes.
 	Host string `json:"host,omitempty"`
 
 	// The path prefix under which to serve all ACME
 	// endpoints. All other requests will not be served
 	// by this handler and will be passed through to
-	// the next one. Default: "/acme/"
+	// the next one. Default: "/acme/".
 	// COMPATIBILITY NOTE / TODO: This property may go away in the
 	// future, as it is currently only required due to
 	// limitations in the underlying library. Do not rely
@@ -92,9 +92,6 @@ func (ash *Handler) Provision(ctx caddy.Context) error {
 	if ash.CA == "" {
 		ash.CA = caddypki.DefaultCAID
 	}
-	if ash.Host == "" {
-		ash.Host = defaultHost
-	}
 	if ash.PathPrefix == "" {
 		ash.PathPrefix = defaultPathPrefix
 	}
@@ -138,17 +135,23 @@ func (ash *Handler) Provision(ctx caddy.Context) error {
 		return err
 	}
 
-	acmeAuth, err := acme.New(auth, acme.AuthorityOptions{
-		DB:     auth.GetDatabase().(nosql.DB),     // stores all the server state
-		DNS:    ash.Host,                          // used for directory links; TODO: not needed
-		Prefix: strings.Trim(ash.PathPrefix, "/"), // used for directory links
-	})
-	if err != nil {
-		return err
+	var acmeDB acme.DB
+	if authorityConfig.DB != nil {
+		acmeDB, err = acmeNoSQL.New(auth.GetDatabase().(nosql.DB))
+		if err != nil {
+			return fmt.Errorf("configuring ACME DB: %v", err)
+		}
 	}
 
 	// create the router for the ACME endpoints
-	acmeRouterHandler := acmeAPI.New(acmeAuth)
+	acmeRouterHandler := acmeAPI.NewHandler(acmeAPI.HandlerOptions{
+		CA:     auth,
+		DB:     acmeDB,                            // stores all the server state
+		DNS:    ash.Host,                          // used for directory links
+		Prefix: strings.Trim(ash.PathPrefix, "/"), // used for directory links
+	})
+
+	// extract its http.Handler so we can use it directly
 	r := chi.NewRouter()
 	r.Route(ash.PathPrefix, func(r chi.Router) {
 		acmeRouterHandler.Route(r)
@@ -212,10 +215,7 @@ func (ash Handler) openDatabase() (*db.AuthDB, error) {
 	return database.(databaseCloser).DB, err
 }
 
-const (
-	defaultHost       = "localhost"
-	defaultPathPrefix = "/acme/"
-)
+const defaultPathPrefix = "/acme/"
 
 var keyCleaner = regexp.MustCompile(`[^\w.-_]`)
 var databasePool = caddy.NewUsagePool()

modules/caddypki/crypto.go

@@ -23,7 +23,7 @@ import (
 	"crypto/x509"
 	"encoding/pem"
 	"fmt"
-	"io/ioutil"
+	"os"
 	"strings"
 )
 
@@ -137,11 +137,11 @@ type KeyPair struct {
 func (kp KeyPair) Load() (*x509.Certificate, interface{}, error) {
 	switch kp.Format {
 	case "", "pem_file":
-		certData, err := ioutil.ReadFile(kp.Certificate)
+		certData, err := os.ReadFile(kp.Certificate)
 		if err != nil {
 			return nil, nil, err
 		}
-		keyData, err := ioutil.ReadFile(kp.PrivateKey)
+		keyData, err := os.ReadFile(kp.PrivateKey)
 		if err != nil {
 			return nil, nil, err
 		}

modules/caddypki/pki.go

@@ -26,10 +26,15 @@ func init() {
 }
 
 // PKI provides Public Key Infrastructure facilities for Caddy.
+//
+// This app can define certificate authorities (CAs) which are capable
+// of signing certificates. Other modules can be configured to use
+// the CAs defined by this app for issuing certificates or getting
+// key information needed for establishing trust.
 type PKI struct {
-	// The CAs to manage. Each CA is keyed by an ID that is used
-	// to uniquely identify it from other CAs. The default CA ID
-	// is "local".
+	// The certificate authorities to manage. Each CA is keyed by an
+	// ID that is used to uniquely identify it from other CAs.
+	// The default CA ID is "local".
 	CAs map[string]*CA `json:"certificate_authorities,omitempty"`
 
 	ctx caddy.Context