Skip to content

Instantly share code, notes, and snippets.

@yusukebe

yusukebe/streaming.ts

Last active April 10, 2024 09:29
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save yusukebe/b701b2d810c92f374b07780f8503782e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save yusukebe/b701b2d810c92f374b07780f8503782e to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. yusukebe revised this gist Apr 4, 2024. 1 changed file with 11 additions and 11 deletions.
    22 changes: 11 additions & 11 deletions streaming.ts
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    import { Hono } from 'hono'
    import { streamText } from 'hono/streaming'
    import { Ai } from '@cloudflare/ai'
    import { events } from 'fetch-event-stream'
    import { streamText } from 'hono/streaming'
    import { Hono } from 'hono'

    type Bindings = {
    AI: any
    @@ -11,18 +11,18 @@ const app = new Hono<{ Bindings: Bindings }>()

    app.get('/', async (c) => {
    const ai = new Ai(c.env.AI)
    const s = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
    prompt: 'What is the origin of the phrase Hello, World',
    const aiStream = (await ai.run('@cf/meta/llama-2-7b-chat-int8', {
    prompt: 'tell me about ramen',
    stream: true
    })
    const res = new Response(s as ReadableStream)
    })) as ReadableStream

    return streamText(c, async (stream) => {
    const chunks = events(res)
    for await (const event of chunks) {
    const obj = JSON.parse(event.data)
    stream.write(obj.response)
    const chunks = events(new Response(aiStream))
    for await (const chunk of chunks) {
    const data = JSON.parse(chunk.data!)
    stream.write(data.response)
    }
    })
    })

    export default app
    export default app
  2. yusukebe created this gist Apr 4, 2024.
    28 changes: 28 additions & 0 deletions streaming.ts
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,28 @@
    import { Ai } from '@cloudflare/ai'
    import { events } from 'fetch-event-stream'
    import { streamText } from 'hono/streaming'
    import { Hono } from 'hono'

    type Bindings = {
    AI: any
    }

    const app = new Hono<{ Bindings: Bindings }>()

    app.get('/', async (c) => {
    const ai = new Ai(c.env.AI)
    const s = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
    prompt: 'What is the origin of the phrase Hello, World',
    stream: true
    })
    const res = new Response(s as ReadableStream)
    return streamText(c, async (stream) => {
    const chunks = events(res)
    for await (const event of chunks) {
    const obj = JSON.parse(event.data)
    stream.write(obj.response)
    }
    })
    })

    export default app