This post extends my formalization of feedback-efficient intelligence with a second-level insight: that the structure of the feedback channel itself must be learned or evolved. In other words, agents must discover not just how to update from feedback, but which feedback matters at all.

1. Definitions

• \( \mathcal{E} \): the environment (a stochastic dynamical system)
• \( \pi \): the agent’s policy
• \( \hat{M}_t \): the internal model of the environment at time \( t \)
• \( \mathcal{F}_t \): feedback received at time \( t \)
• \( U_t \): utility or performance at time \( t \)
• \( I(\mathcal{F}_t; \Delta \hat{M}_t) \): mutual information between feedback and model update
• \( I(\mathcal{E}; \mathcal{F}_t) \): mutual information between the environment and the feedback

2. Meta-Theorem: Feedback Channel Learning

No intelligent system can be feedback-efficient without either possessing or learning a model of the mutual information structure between the environment and its feedback signals:

3. Interpretation

This reflects what evolution did for humans: it sculpted our attention, pain, curiosity, and perception toward signals with consistently high mutual information about task success. It’s not just that we learn from feedback. We evolved to learn what counts as feedback.

This meta-level principle applies to artificial agents as well. In order to sustain feedback-efficient intelligence, they must eventually learn a generative model over the feedback channel. Otherwise, they risk optimizing over noise or reacting to uninformative stimuli. Feedback channels aren't given, they're discovered.