Skip to content

Instantly share code, notes, and snippets.

@warrenronsiek

warrenronsiek/streamLines.scala

Created May 11, 2020 11:26
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save warrenronsiek/fd19fc931299fa4a02e82f5298b4323a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save warrenronsiek/fd19fc931299fa4a02e82f5298b4323a to your computer and use it in GitHub Desktop.
Download ZIP
Stream file lines from S3 in Scala
Raw
streamLines.scala
import com.amazonaws.services.s3.AmazonS3
import com.amazonaws.services.s3.model.{GetObjectRequest, S3Object => JS3Object}
import java.io.{BufferedInputStream, BufferedReader, File}
import java.util.zip.GZIPInputStream
def stream(bucket: String, key: String)(implicit client: AmazonS3): Iterator[String] = {
val js3: JS3Object = client.getObject(new GetObjectRequest(bucket, key))
val s3ObjectInputStream = js3.getObjectContent
val br = new BufferedReader(new BufferedInputStream(new GZIPInputStream(s3ObjectInputStream)).reader)
Stream.continually(br.readLine()).takeWhile(_ != null).toIterator
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment