스트림 프로세싱 with Faust - Table

Faust에는 Table이라는 개념이 있다. Table을 생성하면 스트림데이터와 함께 사용 될 수 있다.

 

from dataclasses import asdict, dataclass
import json
import random

import faust


@dataclass
class ClickEvent(faust.Record):
    email: str
    timestamp: str
    uri: str
    number: int


app = faust.App("exercise6", broker="kafka://localhost:9092")
clickevents_topic = app.topic("com.udacity.streams.clickevents", value_type=ClickEvent)

#
# TODO: Define a uri summary table
#
uri_summary_table = app.Table("uri_summary", default=int)


@app.agent(clickevents_topic)
async def clickevent(clickevents):
    #
    # TODO: Group By URI
    #
    async for ce in clickevents.group_by(ClickEvent.uri):
        #
        # TODO: Use the URI as key, and add the number for each click event
        #
        uri_summary_table[ce.uri] += ce.number
        print(f"{ce.uri}: {uri_summary_table[ce.uri]}")


if __name__ == "__main__":
    app.main()

위 코드에서 'uri_summary'라고하는 table을 생성하였다. 'com.udacity.streams.clickevents'로 부터 받은 json데이터를 토대로 uri를 groupBy로 operation구문을 타도록 한다. 이후에 key/value로서 동일한 uri이 존재한다면 number를 더하도록한다.

 

위 코드의 결과는 아래와 같다.

root@fedf8c3ff7a5:/home/workspace# python exercise6.6.solution.py  worker
┌ƒaµS† v1.7.4─┬──────────────────────────────────────────┐
│ id          │ exercise6                                │
│ transport   │ [URL('kafka://localhost:9092')]          │
│ store       │ memory:                                  │
│ web         │ http://localhost:6066/                   │
│ log         │ -stderr- (warn)                          │
│ pid         │ 832                                      │
│ hostname    │ fedf8c3ff7a5                             │
│ platform    │ CPython 3.7.3 (Linux x86_64)             │
│ drivers     │                                          │
│   transport │ aiokafka=1.0.6                           │
│   web       │ aiohttp=3.6.2                            │
│ datadir     │ /home/workspace/exercise6-data           │
│ appdir      │ /home/workspace/exercise6-data/v1        │
└─────────────┴──────────────────────────────────────────┘
starting➢ ◉
 😊
[2019-11-21 08:58:12,838: WARNING]: https://koch.info/explore/tags/register.php: 672 
[2019-11-21 08:58:12,840: WARNING]: http://www.merritt-reeves.info/category/blog/category/terms/: 829 
[2019-11-21 08:58:12,841: WARNING]: http://rodriguez.com/tags/main/: 516 
[2019-11-21 08:58:12,842: WARNING]: http://hernandez.info/: 922 
[2019-11-21 08:58:12,844: WARNING]: https://www.hancock-lawrence.com/faq/: 760 
[2019-11-21 08:58:12,845: WARNING]: https://vasquez-turner.com/main/app/explore/author/: 729 
[2019-11-21 08:58:12,847: WARNING]: https://www.wilkins.com/blog/search/login.htm: 821 
[2019-11-21 08:58:12,848: WARNING]: http://www.banks.com/faq.htm: 805 
[2019-11-21 08:58:12,849: WARNING]: http://murphy-tran.com/home/: 1236 
[2019-11-21 08:58:12,850: WARNING]: http://williams.info/search.asp: 754 
[2019-11-21 08:58:12,852: WARNING]: http://fry-travis.com/explore/about.html: 203 
[2019-11-21 08:58:12,853: WARNING]: https://www.hernandez.info/categories/privacy.htm: 651 
[2019-11-21 08:58:12,854: WARNING]: https://smith-thomas.org/search/app/category.jsp: 959 
[2019-11-21 08:58:12,855: WARNING]: https://www.york.com/tags/main/privacy.php: 562 
[2019-11-21 08:58:12,856: WARNING]: http://harvey-davis.com/main/tags/register/: 88 
[2019-11-21 08:58:12,857: WARNING]: http://taylor.com/home/: 197 
[2019-11-21 08:58:12,858: WARNING]: https://www.perez-french.biz/category/: 98 
[2019-11-21 08:58:12,859: WARNING]: http://www.christensen.org/homepage/: 755 
[2019-11-21 08:58:12,860: WARNING]: https://www.phillips-christian.com/terms.php: 324 
[2019-11-21 08:58:12,862: WARNING]: http://nelson.biz/login.asp: 202 
[2019-11-21 08:58:12,862: WARNING]: https://taylor.org/: 1150 
[2019-11-21 08:58:12,864: WARNING]: http://davis.com/category/home/: 456 
[2019-11-21 08:58:12,865: WARNING]: http://garcia.net/search.html: 257 
[2019-11-21 08:58:12,865: WARNING]: https://www.baldwin.com/posts/register.html: 54 
[2019-11-21 08:58:12,867: WARNING]: https://www.knight-zuniga.com/privacy/: 293 
[2019-11-21 08:58:12,868: WARNING]: http://www.soto-andrade.biz/about.php: 769 
[2019-11-21 08:58:12,869: WARNING]: http://www.austin.net/app/login/: 904 
[2019-11-21 08:58:12,870: WARNING]: http://www.richardson.com/about/: 305 
[2019-11-21 08:58:12,871: WARNING]: http://davila.biz/index/: 561 
[2019-11-21 08:58:12,873: WARNING]: https://www.curtis-sanders.biz/: 365 
[2019-11-21 08:58:12,874: WARNING]: http://diaz-hughes.net/: 441 
[2019-11-21 08:58:12,875: WARNING]: https://lopez.net/app/categories/category/index/: 91 
[2019-11-21 08:58:12,876: WARNING]: https://chung-roberts.com/wp-content/main/search/privacy/: 352 
[2019-11-21 08:58:12,877: WARNING]: http://simon.com/privacy/: 77 
[2019-11-21 08:58:12,879: WARNING]: https://medina-francis.com/main.html: 424 
[2019-11-21 08:58:12,880: WARNING]: http://www.simmons-shelton.com/search/posts/app/index/: 513 
[2019-11-21 08:58:12,888: WARNING]: https://johnson-palmer.info/main/: 348 
[2019-11-21 08:58:12,890: WARNING]: https://santos-mcmillan.com/: 596 
[2019-11-21 08:58:12,892: WARNING]: https://stanton.com/category/list/search/: 603 
[2019-11-21 08:58:12,893: WARNING]: http://white.com/homepage.htm: 699 
[2019-11-21 08:58:12,895: WARNING]: http://www.johnson.com/category/search/blog/search/: 847 
[2019-11-21 08:58:12,896: WARNING]: https://www.peters.net/explore/wp-content/about.htm: 980 
[2019-11-21 08:58:12,898: WARNING]: https://www.griffith.com/app/tag/list/login.htm: 802 
[2019-11-21 08:58:12,900: WARNING]: http://www.estrada.info/terms.php: 696 
[2019-11-21 08:58:12,902: WARNING]: http://www.harris.info/posts/category/home/: 155 
[2019-11-21 08:58:12,903: WARNING]: http://rosario.com/homepage/: 619 
[2019-11-21 08:58:12,905: WARNING]: https://www.anderson.com/faq/: 1420 
[2019-11-21 08:58:12,906: WARNING]: http://www.dawson-padilla.com/categories/privacy/: 585 
[2019-11-21 08:58:12,918: WARNING]: https://reilly.net/: 864 
[2019-11-21 08:58:12,919: WARNING]: https://www.singleton.com/home/: 347 
[2019-11-21 08:58:12,920: WARNING]: http://www.richardson.com/about/: 1014 
[2019-11-21 08:58:12,929: WARNING]: https://www.edwards.com/faq.htm: 532 
[2019-11-21 08:58:12,938: WARNING]: http://lamb.com/home/: 701 
[2019-11-21 08:58:12,940: WARNING]: https://www.thompson.com/explore/app/post/: 225