AbnormalLemmings
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
kenna@lemmy.dbzer0.com to star technology and research @lemm.ee · 2 years ago

Language Models Represent Space and Time - #AI

arxiv.org

external-link
message-square
0
link
fedilink
1
external-link

Language Models Represent Space and Time - #AI

arxiv.org

kenna@lemmy.dbzer0.com to star technology and research @lemm.ee · 2 years ago
message-square
0
link
fedilink
Language Models Represent Space and Time
arxiv.org
external-link
The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.
alert-triangle
You must log in or register to comment.

star technology and research @lemm.ee

star_adjacent@lemm.ee

Subscribe from Remote Instance

Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !star_adjacent@lemm.ee

generally related/to be applied to design, engineering, or use of socially therapeutic robots

Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 2 users / day
  • 2 users / week
  • 2 users / month
  • 2 users / 6 months
  • 1 local subscriber
  • 56 subscribers
  • 41 Posts
  • 0 Comments
  • Modlog
  • mods:
  • kenna@lemm.ee
  • UI: unknown version
  • BE: 0.19.11
  • Modlog
  • Legal
  • Instances
  • Docs
  • Code
  • join-lemmy.org