Why Run Your Own Video Conferencing Server?
Zoom fatigue is real, but so is Zoom’s privacy policy. If you’re managing a team, a community, or a small business and you’d rather keep video calls off someone else’s infrastructure, Nextcloud Talk gives you a fully self-hosted alternative – video conferencing, screen sharing, and chat running entirely on hardware you control.
What You Need Before You Start
Nextcloud Talk is an app that runs on top of a Nextcloud instance. That means your first requirement is a working Nextcloud installation – version 25 or higher is strongly recommended. If you’re running Nextcloud on a VPS, a home server, or a NAS device, the base setup is essentially the same. You’ll also need a domain name with a valid SSL certificate, because Talk refuses to activate camera and microphone access over plain HTTP. That’s a browser security constraint, not a Nextcloud quirk, and it applies universally.
For small groups – say, two to four people calling at once – your existing Nextcloud server can handle the media streams directly using its built-in WebRTC support. But WebRTC peer-to-peer connections scale poorly. Once you’re hosting calls with five or more participants, the connection web becomes inefficient and call quality degrades noticeably. This is where the High-Performance Backend (HPB) comes in, and where most self-hosters either stop reading the docs or decide to keep calls small. Both choices are valid, but you should know the tradeoff before you commit.
The High-Performance Backend is an open-source signaling server built by the Nextcloud team, written in Go. It handles signaling and can be paired with a Janus WebRTC gateway to offload media routing from your Nextcloud server entirely. Setting this up requires a separate server or at minimum a separate Docker environment – it cannot run cleanly on the same process as Nextcloud without resource conflicts. Plan for at least 2 GB of RAM dedicated to the HPB if you expect more than ten concurrent users. A $6-per-month VPS can handle this load comfortably at low to moderate call volumes.
Before touching any configuration, confirm your firewall rules. Nextcloud Talk requires UDP port 3478 open for STUN/TURN traffic, and if you’re deploying a TURN server (covered below), TCP 5349 as well. Many self-hosters skip this and then spend an hour debugging why audio works but video doesn’t – the answer is almost always a blocked UDP port on the network edge.
Installing Talk and Configuring the TURN Server
Inside your Nextcloud admin panel, navigate to Apps, search for “Talk,” and install it. The app activates immediately and users can start one-on-one calls right away using the peer-to-peer mode. But without a TURN server configured, calls will fail whenever participants are behind strict NAT – which covers most home internet connections and corporate firewalls. TURN (Traversal Using Relays around NAT) acts as a relay, routing media through a known public IP when direct peer connections aren’t possible.
The most widely used TURN server for this setup is Coturn, which is free, open-source, and available in most Linux package repositories. Install it with apt install coturn on Debian or Ubuntu systems. The core configuration lives in /etc/turnserver.conf. At minimum, set your realm to your domain name, define a static-auth-secret (a long random string), and enable use-auth-secret for time-limited credential generation. You’ll also want to set no-multicast-peers and deny-loopback-peers to prevent the TURN server from being used as a proxy for internal network access – a real security exposure if left unchecked.
Once Coturn is running, return to the Nextcloud admin panel and go to Talk settings. Enter your TURN server URL in the format turns:yourdomain.com:5349 – use the turns: prefix for TLS-secured connections, not plain turn:. Paste the same static auth secret you set in Coturn’s config. Nextcloud will display a small green checkmark if the connection test passes. If it shows red, the most likely causes are a firewall blocking port 5349, a mismatched secret, or a certificate error on the TURN server’s TLS config.
For the High-Performance Backend, clone the nextcloud-spreed-signaling repository from GitHub and follow its build instructions. The service expects a configuration file pointing to your Nextcloud server’s URL, an NATS messaging server (also self-hosted, lightweight), and optionally a Janus instance for MCU-based media routing. The signaling server’s config uses TOML format – pay close attention to the [backend] section, where you define which Nextcloud instances are allowed to connect. A misconfigured allowlist is the single most common reason HPB installations fail silently.
Once the signaling server is running, add its URL to Nextcloud’s Talk admin settings under “High-performance backend.” Generate a shared secret and enter it in both the Nextcloud config and the signaling server’s TOML file. At this point, restart both services and create a group call with five or more accounts to verify the HPB is handling the session – you can confirm this by checking the signaling server’s logs for active session entries during the call.
Locking Down Access and Managing Users
Nextcloud Talk inherits the user and group system from your main Nextcloud instance, which means access control is straightforward if your user management is already clean. You can restrict who can create public conversations, who can start calls, and whether guests (unauthenticated users) are allowed to join via shared links. The guest access setting is in Talk’s admin panel under “Allow guests to join conversations.” Leaving this on is useful for client-facing calls but opens your server to anyone with a link, so consider pairing it with conversation passwords for sensitive meetings. If you’re running AdGuard Home as a self-hosted DNS filter on the same network, you can use it to block Nextcloud Talk’s domain from resolving externally if you want to restrict access to LAN-only usage.
Recording is the feature most people ask about and the one Nextcloud Talk handles least gracefully at the self-hosted level. Native recording support requires the recording backend, a separate service that is still experimental as of recent releases. The practical alternative most administrators use is a local screen recorder on the participant’s machine, or a bot account running OBS that joins the call and captures output. Neither is elegant, but recording video conference calls without a dedicated media server is fundamentally a hard problem – Nextcloud isn’t unique in struggling with it.
Testing and What to Watch For
Run a full end-to-end test with accounts on different networks – one on your local LAN and one over a mobile connection or a VPN. This is the only reliable way to catch TURN server failures that wouldn’t show up in a same-network test. Check the browser console for ICE connection errors during the call; a successful TURN relay will show a connection type of “relay” in the WebRTC internals page (chrome://webrtc-internals in Chrome or about:webrtc in Firefox).
The one variable that catches most self-hosters off guard is upload bandwidth. Video conferencing is symmetrical in its demands – your server’s upload speed matters as much as download, and residential internet connections are asymmetric by design. If your Nextcloud server sits behind a home ISP connection with 10 Mbps upload, you’ll hit a ceiling fast with multi-participant calls, even with the HPB handling signaling. A 720p video stream typically consumes 1.5 to 2.5 Mbps per participant. Do the math before your first team call.
